Andrew Boringhttps://andrewboring.com/2020-04-06T00:00:00-04:00Recovering Data From XFS2020-04-06T00:00:00-04:002020-04-06T00:00:00-04:00Andrew Boringtag:andrewboring.com,2020-04-06:/content/xfs-recovery.html<p>Monday morning.<br>
I should stop right there, really. That tells you everything you need to know.</p>
<p>But to continue the narrative, I had to deal with a server whose RAID-1 mirror crapped itself.
It's a legacy server, running some old legacy stuff that hadn't been converted to a more scalable …</p><p>Monday morning.<br>
I should stop right there, really. That tells you everything you need to know.</p>
<p>But to continue the narrative, I had to deal with a server whose RAID-1 mirror crapped itself.
It's a legacy server, running some old legacy stuff that hadn't been converted to a more scalable, cloud-like service. I asked the data center techs to hook up a remote IP-KVM so I could get eyes on it, but Lantronix Java + macOS is an exercise in futility and frustration.</p>
<p>The server was inaccessible from console (using crash cart) when I arrived at the data center.
Rebooted, and found that the main RAID-1 mirror (hosting the primary OS, cPanel, and accounts) was marked as a "Foreign" configuration by the Dell PERC card.</p>
<p>Entered system RAID utility. Disk 2 was marked as offline, and configuration was marked "Foreign" instead of "Degraded".</p>
<p>I removed failed Disk 2 drive, replaced with new Disk 2 from another server, but it wasn't similar enough for the PERC card to allow it to be used as replacement to rebuild the mirror. Still marked "Foreign".</p>
<p>I tried re-seating the original Disk 2, and eventually found the "Clear Foreign Configuration" option (not to be confused with the more prominent "Clear Configuration" option, which wipes everything including the other RAID mirrors installed on the server). It started "rebuilding" disk 1. WTF? I don't want to rebuild Disk 1 from failed Disk 2!</p>
<p>Reboot, since there's no "stop" button.</p>
<p>Recreated mirror with just Disk 1, but did not initialize.</p>
<p>Rebooted and it booted up into OS/cPanel, but apparently the files were dated from 2017 and Disk 1 was not being mirrored since then. My assumption is that the failed disk 2 was receiving all the writes from the RAID controller and had the current files.</p>
<p>Anyway, time to recover.</p>
<hr>
<p>High-level steps:</p>
<ol>
<li>Use xfs_repair -n to check without modification, like fsck -n.</li>
<li>Try to use xfs_repair to fix.</li>
<li>If prompted to mount the partition to replay the log, unmount, and then re-run xfs_repair, try that. But you're probably in this situation because you <em>can't</em> mount the filesystem.</li>
<li>If prompted to run with -L to delete the log (a destructive, last-resort operation), try to test it on a copy of the metadata first.<ul>
<li>xfs_metadump [partition] /path/to/file.metadump</li>
<li>xfs_mdrestore /path/to/file.metadump /path/to/file.img</li>
<li>losetup --show --find /path/to/file.img</li>
<li>xfs_repair -L /dev/loop0</li>
<li>mount /dev/loop0 /mnt</li>
<li>check the damage</li>
<li>(note, this is an image of file system layout, but not the actual data)</li>
</ul>
</li>
<li>Finally, copy the partition data to an image file using ddrescue, and then try to repair the copy.<ul>
<li>ddrescue -f -n [partition] /path/to/rescued.img rescue.log</li>
<li>ddrescue -d -f -r3 [partition] /path/to/rescued.img rescue.log</li>
<li>losetup --show --find /path/to/rescued.img</li>
<li>xfs_repair -L /dev/loop0</li>
<li>mount /dev/loop0 /mnt</li>
<li>check the damage</li>
</ul>
</li>
<li>If that doesn't work, you can try xfs_repair -L directly on the affected partition, but not before you pray to your deity of choice.</li>
</ol>
<p>Since macOS sucks for working with non-Apple partitions/data, I used a Raspberry Pi 3B+ running Arch Linux. I connected the SATA drive to a SATA-to-USB adapter, and later attached a second external USB drive for copying data to.</p>
<p>Plugged in the bad drive first. Registers as /dev/sda.
Take a look at the partitions and try to mount it:</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">fdisk</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda</span><span class="w"></span>
<span class="k">Disk</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="nl">sda</span><span class="p">:</span><span class="w"> </span><span class="mf">149.5</span><span class="w"> </span><span class="n">GiB</span><span class="p">,</span><span class="w"> </span><span class="mi">160041885696</span><span class="w"> </span><span class="n">bytes</span><span class="p">,</span><span class="w"> </span><span class="mi">312581808</span><span class="w"> </span><span class="n">sectors</span><span class="w"></span>
<span class="k">Disk</span><span class="w"> </span><span class="nl">model</span><span class="p">:</span><span class="w"> </span><span class="mi">00</span><span class="n">AAJS</span><span class="o">-</span><span class="mi">60</span><span class="n">PSA0</span><span class="w"> </span>
<span class="nl">Units</span><span class="p">:</span><span class="w"> </span><span class="n">sectors</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="n">bytes</span><span class="w"></span>
<span class="n">Sector</span><span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="p">(</span><span class="n">logical</span><span class="o">/</span><span class="n">physical</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="n">bytes</span><span class="w"></span>
<span class="n">I</span><span class="o">/</span><span class="n">O</span><span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="p">(</span><span class="n">minimum</span><span class="o">/</span><span class="n">optimal</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="n">bytes</span><span class="w"></span>
<span class="n">Disklabel</span><span class="w"> </span><span class="nl">type</span><span class="p">:</span><span class="w"> </span><span class="n">dos</span><span class="w"></span>
<span class="k">Disk</span><span class="w"> </span><span class="nl">identifier</span><span class="p">:</span><span class="w"> </span><span class="mh">0x00070f16</span><span class="w"></span>
<span class="n">Device</span><span class="w"> </span><span class="n">Boot</span><span class="w"> </span><span class="k">Start</span><span class="w"> </span><span class="k">End</span><span class="w"> </span><span class="n">Sectors</span><span class="w"> </span><span class="k">Size</span><span class="w"> </span><span class="n">Id</span><span class="w"> </span><span class="n">Type</span><span class="w"></span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda1</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="mi">1026047</span><span class="w"> </span><span class="mi">1024000</span><span class="w"> </span><span class="mi">500</span><span class="n">M</span><span class="w"> </span><span class="mi">83</span><span class="w"> </span><span class="n">Linux</span><span class="w"></span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda2</span><span class="w"> </span><span class="mi">1026048</span><span class="w"> </span><span class="mi">311427071</span><span class="w"> </span><span class="mi">310401024</span><span class="w"> </span><span class="mi">148</span><span class="n">G</span><span class="w"> </span><span class="mi">8</span><span class="n">e</span><span class="w"> </span><span class="n">Linux</span><span class="w"> </span><span class="n">LVM</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sda2</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="w"></span>
<span class="nl">mount</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="nl">mnt</span><span class="p">:</span><span class="w"> </span><span class="k">unknown</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="s1">'LVM2_member'</span><span class="p">.</span><span class="w"></span>
</pre></div>
<p>Oh, right. LVM volume. Duh.
Let's install LVM tools.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">pacman</span><span class="w"> </span><span class="o">-</span><span class="n">S</span><span class="w"> </span><span class="n">lvm2</span><span class="w"></span>
<span class="n">resolving</span><span class="w"> </span><span class="n">dependencies</span><span class="p">...</span><span class="w"></span>
<span class="n">looking</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">conflicting</span><span class="w"> </span><span class="n">packages</span><span class="p">...</span><span class="w"></span>
<span class="n">Packages</span><span class="w"> </span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="n">libaio</span><span class="o">-</span><span class="mf">0.3.112</span><span class="o">-</span><span class="mi">2</span><span class="w"> </span><span class="n">thin</span><span class="o">-</span><span class="n">provisioning</span><span class="o">-</span><span class="n">tools</span><span class="o">-</span><span class="mf">0.8.5</span><span class="o">-</span><span class="mi">3</span><span class="w"> </span><span class="n">lvm2</span><span class="o">-</span><span class="mf">2.02.186</span><span class="o">-</span><span class="mi">5</span><span class="w"></span>
<span class="n">Total</span><span class="w"> </span><span class="n">Download</span><span class="w"> </span><span class="k">Size</span><span class="err">:</span><span class="w"> </span><span class="mf">1.51</span><span class="w"> </span><span class="n">MiB</span><span class="w"></span>
<span class="n">Total</span><span class="w"> </span><span class="n">Installed</span><span class="w"> </span><span class="k">Size</span><span class="err">:</span><span class="w"> </span><span class="mf">7.30</span><span class="w"> </span><span class="n">MiB</span><span class="w"></span>
<span class="o">::</span><span class="w"> </span><span class="n">Proceed</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">installation</span><span class="vm">?</span><span class="w"> </span><span class="o">[</span><span class="n">Y/n</span><span class="o">]</span><span class="w"> </span><span class="n">y</span><span class="w"></span>
<span class="o">::</span><span class="w"> </span><span class="n">Retrieving</span><span class="w"> </span><span class="n">packages</span><span class="p">...</span><span class="w"></span>
<span class="w"> </span><span class="n">libaio</span><span class="o">-</span><span class="mf">0.3.112</span><span class="o">-</span><span class="mi">2</span><span class="o">-</span><span class="n">armv7h</span><span class="w"> </span><span class="mf">6.0</span><span class="w"> </span><span class="n">KiB</span><span class="w"> </span><span class="mi">604</span><span class="w"> </span><span class="n">KiB</span><span class="o">/</span><span class="n">s</span><span class="w"> </span><span class="mi">00</span><span class="err">:</span><span class="mi">00</span><span class="w"> </span><span class="o">[</span><span class="n">########################################################</span><span class="o">]</span><span class="w"> </span><span class="mi">100</span><span class="o">%</span><span class="w"></span>
<span class="w"> </span><span class="n">thin</span><span class="o">-</span><span class="n">provisioning</span><span class="o">-</span><span class="n">tools</span><span class="o">-</span><span class="mf">0.8.5</span><span class="o">-</span><span class="mi">3</span><span class="o">-</span><span class="n">armv7h</span><span class="w"> </span><span class="mf">326.1</span><span class="w"> </span><span class="n">KiB</span><span class="w"> </span><span class="mf">3.18</span><span class="w"> </span><span class="n">MiB</span><span class="o">/</span><span class="n">s</span><span class="w"> </span><span class="mi">00</span><span class="err">:</span><span class="mi">00</span><span class="w"> </span><span class="o">[</span><span class="n">########################################################</span><span class="o">]</span><span class="w"> </span><span class="mi">100</span><span class="o">%</span><span class="w"></span>
<span class="nl">error</span><span class="p">:</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="n">retrieving</span><span class="w"> </span><span class="k">file</span><span class="w"> </span><span class="s1">'lvm2-2.02.186-5-armv7h.pkg.tar.xz'</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">fl</span><span class="p">.</span><span class="n">us</span><span class="p">.</span><span class="n">mirror</span><span class="p">.</span><span class="n">archlinuxarm</span><span class="p">.</span><span class="n">org</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">requested</span><span class="w"> </span><span class="n">URL</span><span class="w"> </span><span class="n">returned</span><span class="w"> </span><span class="nl">error</span><span class="p">:</span><span class="w"> </span><span class="mi">404</span><span class="w"></span>
<span class="nl">warning</span><span class="p">:</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">retrieve</span><span class="w"> </span><span class="ow">some</span><span class="w"> </span><span class="n">files</span><span class="w"></span>
<span class="nl">error</span><span class="p">:</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">commit</span><span class="w"> </span><span class="k">transaction</span><span class="w"> </span><span class="p">(</span><span class="n">failed</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">retrieve</span><span class="w"> </span><span class="ow">some</span><span class="w"> </span><span class="n">files</span><span class="p">)</span><span class="w"></span>
<span class="n">Errors</span><span class="w"> </span><span class="n">occurred</span><span class="p">,</span><span class="w"> </span><span class="k">no</span><span class="w"> </span><span class="n">packages</span><span class="w"> </span><span class="n">were</span><span class="w"> </span><span class="n">upgraded</span><span class="p">.</span><span class="w"></span>
</pre></div>
<p>Of course. It's Arch. Gotta upgrade the OS first.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# pacman -Syu</span>
<span class="err">:: Synchronizing package databases...</span>
<span class="err">...</span>
<span class="err">:: Retrieving packages...</span>
</pre></div>
<p>Let's find the logical volumes on this LVM partition and activate them:</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">lvscan</span><span class="w"></span>
<span class="w"> </span><span class="nl">WARNING</span><span class="p">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">connect</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">lvmetad</span><span class="p">.</span><span class="w"> </span><span class="n">Falling</span><span class="w"> </span><span class="n">back</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">scanning</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="n">inactive</span><span class="w"> </span><span class="s1">'/dev/centos/swap'</span><span class="w"> </span><span class="o">[</span><span class="n">7.88 GiB</span><span class="o">]</span><span class="w"> </span><span class="n">inherit</span><span class="w"></span>
<span class="w"> </span><span class="n">inactive</span><span class="w"> </span><span class="s1">'/dev/centos/home'</span><span class="w"> </span><span class="o">[</span><span class="n">90.12 GiB</span><span class="o">]</span><span class="w"> </span><span class="n">inherit</span><span class="w"></span>
<span class="w"> </span><span class="n">inactive</span><span class="w"> </span><span class="s1">'/dev/centos/root'</span><span class="w"> </span><span class="o">[</span><span class="n">50.00 GiB</span><span class="o">]</span><span class="w"> </span><span class="n">inherit</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">vgchange</span><span class="w"> </span><span class="o">-</span><span class="n">ay</span><span class="w"></span>
<span class="w"> </span><span class="nl">WARNING</span><span class="p">:</span><span class="w"> </span><span class="n">Failed</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">connect</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">lvmetad</span><span class="p">.</span><span class="w"> </span><span class="n">Falling</span><span class="w"> </span><span class="n">back</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">device</span><span class="w"> </span><span class="n">scanning</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">logical</span><span class="w"> </span><span class="n">volume</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">volume</span><span class="w"> </span><span class="k">group</span><span class="w"> </span><span class="ss">"centos"</span><span class="w"> </span><span class="n">now</span><span class="w"> </span><span class="n">active</span><span class="w"></span>
</pre></div>
<p>Try to mount the root partition first:</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# mount /dev/centos/root /mnt</span>
<span class="err">[root@alarmpi ~]# ls /mnt</span>
<span class="err">aquota.group backup boot error_log home lib media opt quota.group razor-agent.log run scripts sys usr</span>
<span class="err">aquota.user bin dev etc home2 lib64 mnt proc quota.user root sbin srv tmp var</span>
</pre></div>
<p>Success!
Home is a different LVM2 volume, mounted onto the /home directory on the original disk.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">home</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"> </span><span class="o">/</span><span class="n">server6</span><span class="o">/</span><span class="n">home</span><span class="w"></span>
<span class="nl">mount</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">server6</span><span class="o">/</span><span class="nl">home</span><span class="p">:</span><span class="w"> </span><span class="n">mount</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="k">system</span><span class="w"> </span><span class="k">call</span><span class="w"> </span><span class="nl">failed</span><span class="p">:</span><span class="w"> </span><span class="k">No</span><span class="w"> </span><span class="k">data</span><span class="w"> </span><span class="n">available</span><span class="p">.</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">home</span><span class="o">*</span><span class="w"></span>
<span class="n">home</span><span class="o">/</span><span class="w"> </span><span class="n">home2</span><span class="o">/</span><span class="w"></span>
</pre></div>
<p>Maybe mount it directly on the Pi, rather than inside the mounted root partition?</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">home3</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">home3</span><span class="w"></span>
<span class="nl">mount</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="nl">home3</span><span class="p">:</span><span class="w"> </span><span class="n">mount</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="k">system</span><span class="w"> </span><span class="k">call</span><span class="w"> </span><span class="nl">failed</span><span class="p">:</span><span class="w"> </span><span class="k">No</span><span class="w"> </span><span class="k">data</span><span class="w"> </span><span class="n">available</span><span class="p">.</span><span class="w"></span>
</pre></div>
<p>Nope.
Well, the databases are more important, since a lot of the /home directories are just web content, and that's stored in version control. Let's check /var/lib/mysql on the root partition.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# cd /mnt/var/lib/mysql/</span>
<span class="err">[root@alarmpi mysql]# ls</span>
<span class="err">RPM_UPGRADE_HISTORY user2_wp modsec user11_aclstudygroup user11_linct_prod user11_sddocs user0_prod</span>
<span class="err">RPM_UPGRADE_MARKER-LAST user3_production mysql user11_aclstudygroup_vanilla user11_magedemo user11_staging user0_redesign</span>
<span class="err">auto.cnf user3_test mysql.sock user11_share user11_marketingplaybook user11_staging_old user0_staging</span>
<span class="err">user1_brody user6_prod mysql_upgrade_info user11_bddocs user11_minerva user11_surge user0_test</span>
<span class="err">user1_gasexcrimes ib_logfile0 user11_pack688 user11_captainplanet user11_mcorp user11_unboundary</span>
<span class="err">user1_prod ib_logfile1 performance_schema user11_citysaga user11_pbndocs user11_wtefdocs</span>
<span class="err">cphulkd ibdata1 user11_cmp user11_playbacknow user5_prod</span>
<span class="err">user3_dev server6.example.com.err user11_edc user11_prod user0_dev</span>
<span class="err">user3_prod server6.example.com.pid roundcube user11_user10 user11_prod_wordpress user0_drupal</span>
<span class="err">user0_survey leechprotect user11_aboutplay user11_dev user11_scrum user0_main</span>
</pre></div>
<p>Sweet! Poked around a bit to look at file sizes (no 0 bytes fiiles, etc). Things look good.
Let's rsync it off directly to a network storage device...</p>
<div class="highlight"><pre><span></span><span class="n">root</span><span class="p">@</span><span class="n">alarmpi</span> <span class="n">mysql</span><span class="p">]</span><span class="err">#</span> <span class="n">rsync</span> <span class="o">-</span><span class="n">avz</span> <span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">mysql</span> <span class="n">bkup</span><span class="mf">@192.168.1.15</span><span class="o">:/</span><span class="n">mnt</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">bkup</span><span class="o">/</span><span class="n">server6</span><span class="o">/</span>
<span class="n">This</span> <span class="n">rsync</span> <span class="n">lacks</span> <span class="n">old</span><span class="o">-</span><span class="n">style</span> <span class="o">--</span><span class="n">compress</span> <span class="n">due</span> <span class="n">to</span> <span class="n">its</span> <span class="n">external</span> <span class="n">zlib</span><span class="p">.</span> <span class="n">Try</span> <span class="o">-</span><span class="n">zz</span><span class="p">.</span>
<span class="n">Continuing</span> <span class="n">without</span> <span class="n">compression</span><span class="p">.</span>
<span class="n">The</span> <span class="n">authenticity</span> <span class="n">of</span> <span class="n">host</span> <span class="err">'</span><span class="mf">192.168.1.15</span> <span class="p">(</span><span class="mf">192.168.1.15</span><span class="p">)</span><span class="err">'</span> <span class="n">can</span><span class="err">'</span><span class="n">t</span> <span class="n">be</span> <span class="n">established</span><span class="p">.</span>
<span class="n">ECDSA</span> <span class="n">key</span> <span class="n">fingerprint</span> <span class="n">is</span> <span class="nl">SHA256</span><span class="p">:</span><span class="n">vFpXHAorzqVTuQKSL2WyP5EtE</span><span class="o">+</span><span class="n">MwKPyT1Xij1Hz1fCA</span><span class="p">.</span>
<span class="n">Are</span> <span class="n">you</span> <span class="n">sure</span> <span class="n">you</span> <span class="n">want</span> <span class="n">to</span> <span class="k">continue</span> <span class="n">connecting</span> <span class="p">(</span><span class="n">yes</span><span class="o">/</span><span class="n">no</span><span class="o">/</span><span class="p">[</span><span class="n">fingerprint</span><span class="p">])</span><span class="o">?</span> <span class="n">yes</span>
<span class="nl">Warning</span><span class="p">:</span> <span class="n">Permanently</span> <span class="n">added</span> <span class="err">'</span><span class="mf">192.168.1.15</span><span class="err">'</span> <span class="p">(</span><span class="n">ECDSA</span><span class="p">)</span> <span class="n">to</span> <span class="n">the</span> <span class="n">list</span> <span class="n">of</span> <span class="n">known</span> <span class="n">hosts</span><span class="p">.</span>
<span class="n">bkup</span><span class="mf">@192.168.1.15</span><span class="err">'</span><span class="n">s</span> <span class="nl">password</span><span class="p">:</span>
<span class="n">sending</span> <span class="n">incremental</span> <span class="n">file</span> <span class="n">list</span>
<span class="n">created</span> <span class="n">directory</span> <span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">bkup</span><span class="o">/</span><span class="n">server6</span>
<span class="n">mysql</span><span class="o">/</span>
<span class="n">mysql</span><span class="o">/</span><span class="n">RPM_UPGRADE_HISTORY</span>
<span class="n">mysql</span><span class="o">/</span><span class="n">RPM_UPGRADE_MARKER</span><span class="o">-</span><span class="n">LAST</span>
<span class="n">mysql</span><span class="o">/</span><span class="k">auto</span><span class="p">.</span><span class="n">cnf</span>
<span class="p">...</span>
<span class="p">...</span>
</pre></div>
<p>Now, let's install the packages to get xfs_repair on Arch.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# pacman -S xfsprogs</span>
<span class="err">resolving dependencies...</span>
<span class="err">looking for conflicting packages...</span>
</pre></div>
<p>Let's try to mount it again.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# mount /dev/centos/home /mnt/home</span>
<span class="c">mount: /mnt/home: mount(2) system call failed: No data available.</span>
</pre></div>
<p>And run xfs_repair -n to see what would happen (edited for brevity).</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_repair</span><span class="w"> </span><span class="o">-</span><span class="n">n</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">find</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">verify</span><span class="w"> </span><span class="n">superblock</span><span class="p">...</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">internal</span><span class="w"> </span><span class="nf">log</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="nf">log</span><span class="p">...</span><span class="w"></span>
<span class="nl">ALERT</span><span class="p">:</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">has</span><span class="w"> </span><span class="n">valuable</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="n">changes</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">which</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">being</span><span class="w"></span>
<span class="n">ignored</span><span class="w"> </span><span class="n">because</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="o">-</span><span class="n">n</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="n">was</span><span class="w"> </span><span class="n">used</span><span class="p">.</span><span class="w"> </span><span class="n">Expect</span><span class="w"> </span><span class="n">spurious</span><span class="w"> </span><span class="n">inconsistencies</span><span class="w"></span>
<span class="n">which</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">resolved</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="k">first</span><span class="w"> </span><span class="n">mounting</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">replay</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">scan</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">freespace</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">inode</span><span class="w"> </span><span class="n">maps</span><span class="p">...</span><span class="w"></span>
<span class="n">sb_fdblocks</span><span class="w"> </span><span class="mi">1723017</span><span class="p">,</span><span class="w"> </span><span class="n">counted</span><span class="w"> </span><span class="mi">1731714</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">found</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">inode</span><span class="w"> </span><span class="n">chunk</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="k">each</span><span class="w"> </span><span class="n">AG</span><span class="p">...</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">scan</span><span class="w"> </span><span class="p">(</span><span class="n">but</span><span class="w"> </span><span class="n">don</span><span class="s1">'t clear) agi unlinked lists...</span>
<span class="s1"> - process known inodes and perform inode discovery...</span>
<span class="s1"> - agno = 0</span>
<span class="s1">xfs_repair: read failed: No data available</span>
<span class="s1">bad magic number 0x6d65 on inode 5840900</span>
<span class="s1">bad version number 0x6f on inode 5840900</span>
<span class="s1">bad next_unlinked 0x2f5a656e on inode 5840900</span>
<span class="s1">bad magic number 0x2f68 on inode 5840906, would reset magic number</span>
<span class="s1">bad version number 0x65 on inode 5840906, would reset version number</span>
<span class="s1">bad next_unlinked 0x6172792f on inode 5840906, would reset next_unlinked</span>
<span class="s1">bad inode format in inode 5840927</span>
<span class="s1">would have cleared inode 5840927</span>
<span class="s1"> - agno = 1</span>
<span class="s1"> - agno = 2</span>
<span class="s1"> - agno = 3</span>
<span class="s1"> - process newly discovered inodes...</span>
<span class="s1">Phase 4 - check for duplicate blocks...</span>
<span class="s1"> - setting up duplicate extent list...</span>
<span class="s1"> - check for inodes claiming duplicate blocks...</span>
<span class="s1"> - agno = 0</span>
<span class="s1">Phase 7 - verify link counts...</span>
<span class="s1">xfs_repair: read failed: No data available</span>
<span class="s1">xfs_imap_to_bp: xfs_trans_read_buf() returned error -61.</span>
<span class="s1">couldn'</span><span class="n">t</span><span class="w"> </span><span class="k">map</span><span class="w"> </span><span class="n">inode</span><span class="w"> </span><span class="mi">5840917</span><span class="p">,</span><span class="w"> </span><span class="n">err</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">61</span><span class="p">,</span><span class="w"> </span><span class="n">can</span><span class="s1">'t compare link counts</span>
<span class="s1">xfs_repair: read failed: No data available</span>
<span class="s1">xfs_imap_to_bp: xfs_trans_read_buf() returned error -61.</span>
<span class="s1">couldn'</span><span class="n">t</span><span class="w"> </span><span class="k">map</span><span class="w"> </span><span class="n">inode</span><span class="w"> </span><span class="mi">5840925</span><span class="p">,</span><span class="w"> </span><span class="n">err</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">61</span><span class="p">,</span><span class="w"> </span><span class="n">can</span><span class="err">'</span><span class="n">t</span><span class="w"> </span><span class="n">compare</span><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="n">counts</span><span class="w"></span>
<span class="k">No</span><span class="w"> </span><span class="k">modify</span><span class="w"> </span><span class="n">flag</span><span class="w"> </span><span class="k">set</span><span class="p">,</span><span class="w"> </span><span class="n">skipping</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">flush</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">exiting</span><span class="p">.</span><span class="w"></span>
</pre></div>
<p>Well, there were some I/O errors in my output, so I'm not holding my breath for the repair:</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_repair</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">find</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">verify</span><span class="w"> </span><span class="n">superblock</span><span class="p">...</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">internal</span><span class="w"> </span><span class="nf">log</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="nf">log</span><span class="p">...</span><span class="w"></span>
<span class="nl">ERROR</span><span class="p">:</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">has</span><span class="w"> </span><span class="n">valuable</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="n">changes</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">which</span><span class="w"> </span><span class="n">needs</span><span class="w"> </span><span class="k">to</span><span class="w"></span>
<span class="n">be</span><span class="w"> </span><span class="n">replayed</span><span class="p">.</span><span class="w"> </span><span class="n">Mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">replay</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="p">,</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">unmount</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">before</span><span class="w"></span>
<span class="n">re</span><span class="o">-</span><span class="n">running</span><span class="w"> </span><span class="n">xfs_repair</span><span class="p">.</span><span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="k">are</span><span class="w"> </span><span class="n">unable</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="p">,</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="k">use</span><span class="w"></span>
<span class="n">the</span><span class="w"> </span><span class="o">-</span><span class="n">L</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">destroy</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">attempt</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">repair</span><span class="p">.</span><span class="w"></span>
<span class="n">Note</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">destroying</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">cause</span><span class="w"> </span><span class="n">corruption</span><span class="w"> </span><span class="c1">-- please attempt a mount</span>
<span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">before</span><span class="w"> </span><span class="n">doing</span><span class="w"> </span><span class="n">this</span><span class="p">.</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="n">xfs</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">home</span><span class="w"></span>
<span class="nl">mount</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="nl">home</span><span class="p">:</span><span class="w"> </span><span class="n">mount</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="k">system</span><span class="w"> </span><span class="k">call</span><span class="w"> </span><span class="nl">failed</span><span class="p">:</span><span class="w"> </span><span class="k">No</span><span class="w"> </span><span class="k">data</span><span class="w"> </span><span class="n">available</span><span class="p">.</span><span class="w"></span>
</pre></div>
<p>Yeah. Not so great. I already know I can't mount the partition to replay the log, and the -L switch is a destructive operation.</p>
<p>Let's unmount the root filesystem for a moment and attach a second, external USB drive to copy home data off to. We'll put it onto '/mnt' instead.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n"> 4451.182458</span><span class="o">]</span><span class="w"> </span><span class="n">sd</span><span class="w"> </span><span class="mi">1</span><span class="err">:</span><span class="mi">0</span><span class="err">:</span><span class="mi">0</span><span class="err">:</span><span class="mi">0</span><span class="err">:</span><span class="w"> </span><span class="o">[</span><span class="n">sdb</span><span class="o">]</span><span class="w"> </span><span class="n">Attached</span><span class="w"> </span><span class="n">SCSI</span><span class="w"> </span><span class="k">disk</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">umount</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb1</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="w"></span>
</pre></div>
<p>First, we'll dump the metadata and restore it to a disk image on the USB drive.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_metadump</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">metadump</span><span class="w"></span>
<span class="nl">xfs_metadump</span><span class="p">:</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="nl">failed</span><span class="p">:</span><span class="w"> </span><span class="k">Input</span><span class="o">/</span><span class="k">output</span><span class="w"> </span><span class="n">error</span><span class="w"></span>
<span class="nl">xfs_metadump</span><span class="p">:</span><span class="w"> </span><span class="n">cannot</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="n">inode</span><span class="w"> </span><span class="n">block</span><span class="w"> </span><span class="mi">0</span><span class="o">/</span><span class="mi">365056</span><span class="w"></span>
<span class="nl">xfs_metadump</span><span class="p">:</span><span class="w"> </span><span class="nl">Warning</span><span class="p">:</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">recovery</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">obfuscated</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">leak</span><span class="w"> </span><span class="n">unobfuscated</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="ow">and</span><span class="o">/</span><span class="ow">or</span><span class="w"> </span><span class="n">cause</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="n">corruption</span><span class="p">.</span><span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="n">possible</span><span class="p">,</span><span class="w"> </span><span class="n">please</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">clean</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="p">,</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="n">disable</span><span class="w"> </span><span class="n">obfuscation</span><span class="p">.</span><span class="w"></span>
<span class="nl">cache_purge</span><span class="p">:</span><span class="w"> </span><span class="n">shake</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="n">cache</span><span class="w"> </span><span class="mh">0xfe5320</span><span class="w"> </span><span class="nf">left</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">nodes</span><span class="err">!</span><span class="vm">?</span><span class="w"></span>
<span class="nl">cache_purge</span><span class="p">:</span><span class="w"> </span><span class="n">shake</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="n">cache</span><span class="w"> </span><span class="mh">0xfe5320</span><span class="w"> </span><span class="nf">left</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">nodes</span><span class="err">!</span><span class="vm">?</span><span class="w"></span>
<span class="nl">cache_zero_check</span><span class="p">:</span><span class="w"> </span><span class="n">refcount</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="p">(</span><span class="n">node</span><span class="o">=</span><span class="mh">0x17820c8</span><span class="p">)</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">-</span><span class="n">l</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">metadump</span><span class="w"> </span>
<span class="o">-</span><span class="n">rwxr</span><span class="o">-</span><span class="n">xr</span><span class="o">-</span><span class="n">x</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="n">root</span><span class="w"> </span><span class="mi">844619776</span><span class="w"> </span><span class="n">Apr</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="mi">18</span><span class="err">:</span><span class="mi">48</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">metadump</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_mdrestore</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">metadump</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">img</span><span class="w"></span>
<span class="nl">xfs_mdrestore</span><span class="p">:</span><span class="w"> </span><span class="n">cannot</span><span class="w"> </span><span class="k">set</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="k">size</span><span class="err">:</span><span class="w"> </span><span class="k">File</span><span class="w"> </span><span class="n">too</span><span class="w"> </span><span class="k">large</span><span class="w"></span>
</pre></div>
<p>Oh yeah, the USB drive is FAT32. Can't handle large image files (4GB is the limit).
Wipe it and format it as ext4.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">umount</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mkfs</span><span class="p">.</span><span class="n">ext4</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb1</span><span class="w"></span>
<span class="n">mke2fs</span><span class="w"> </span><span class="mf">1.45.6</span><span class="w"> </span><span class="p">(</span><span class="mi">20</span><span class="o">-</span><span class="n">Mar</span><span class="o">-</span><span class="mi">2020</span><span class="p">)</span><span class="w"></span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb1</span><span class="w"> </span><span class="k">contains</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">vfat</span><span class="w"> </span><span class="k">file</span><span class="w"> </span><span class="k">system</span><span class="w"> </span><span class="n">labelled</span><span class="w"> </span><span class="s1">'UNTITLED'</span><span class="w"></span>
<span class="n">Proceed</span><span class="w"> </span><span class="n">anyway</span><span class="vm">?</span><span class="w"> </span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">N</span><span class="p">)</span><span class="w"> </span><span class="n">y</span><span class="w"></span>
<span class="n">Creating</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="mi">61049645</span><span class="w"> </span><span class="mi">4</span><span class="n">k</span><span class="w"> </span><span class="n">blocks</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="mi">15269888</span><span class="w"> </span><span class="n">inodes</span><span class="w"></span>
<span class="n">Filesystem</span><span class="w"> </span><span class="nl">UUID</span><span class="p">:</span><span class="w"> </span><span class="mf">50818e1</span><span class="n">c</span><span class="o">-</span><span class="n">e8a9</span><span class="o">-</span><span class="mi">42</span><span class="n">a9</span><span class="o">-</span><span class="n">a126</span><span class="o">-</span><span class="mi">405</span><span class="n">f9d5fa8a0</span><span class="w"></span>
<span class="n">Superblock</span><span class="w"> </span><span class="n">backups</span><span class="w"> </span><span class="n">stored</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="nl">blocks</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="mi">32768</span><span class="p">,</span><span class="w"> </span><span class="mi">98304</span><span class="p">,</span><span class="w"> </span><span class="mi">163840</span><span class="p">,</span><span class="w"> </span><span class="mi">229376</span><span class="p">,</span><span class="w"> </span><span class="mi">294912</span><span class="p">,</span><span class="w"> </span><span class="mi">819200</span><span class="p">,</span><span class="w"> </span><span class="mi">884736</span><span class="p">,</span><span class="w"> </span><span class="mi">1605632</span><span class="p">,</span><span class="w"> </span><span class="mi">2654208</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">4096000</span><span class="p">,</span><span class="w"> </span><span class="mi">7962624</span><span class="p">,</span><span class="w"> </span><span class="mi">11239424</span><span class="p">,</span><span class="w"> </span><span class="mi">20480000</span><span class="p">,</span><span class="w"> </span><span class="mi">23887872</span><span class="w"></span>
<span class="n">Allocating</span><span class="w"> </span><span class="k">group</span><span class="w"> </span><span class="nl">tables</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="w"> </span>
<span class="n">Writing</span><span class="w"> </span><span class="n">inode</span><span class="w"> </span><span class="nl">tables</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="w"> </span>
<span class="n">Creating</span><span class="w"> </span><span class="n">journal</span><span class="w"> </span><span class="p">(</span><span class="mi">262144</span><span class="w"> </span><span class="n">blocks</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="n">done</span><span class="w"></span>
<span class="n">Writing</span><span class="w"> </span><span class="n">superblocks</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">accounting</span><span class="w"> </span><span class="nl">information</span><span class="p">:</span><span class="w"> </span><span class="n">done</span><span class="w"> </span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">sdb1</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="w"></span>
</pre></div>
<p>Now, let's try this again.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# xfs_metadump /dev/centos/home /mnt/server6-home.metadata</span>
<span class="c">xfs_metadump: read failed: Input/output error</span>
<span class="c">xfs_metadump: cannot read inode block 0/365056</span>
<span class="c">xfs_metadump: Warning: log recovery of an obfuscated metadata image can leak unobfuscated metadata and/or cause image corruption. If possible, please mount the filesystem to clean the log, or disable obfuscation.</span>
<span class="c">cache_purge: shake on cache 0x1cb6320 left 1 nodes!?</span>
<span class="c">cache_purge: shake on cache 0x1cb6320 left 1 nodes!?</span>
<span class="c">cache_zero_check: refcount is 1, not zero (node=0x24520c8)</span>
</pre></div>
<p>That <code>read failed: Input/Output</code> error doesn't sound good, but let's try to restore the metadata to an image file anyway.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# xfs_mdrestore /mnt/server6-home.metadata /mnt/server6-home.img</span>
<span class="err">[root@alarmpi ~]# xfs_repair /mnt/server6-home.img</span>
<span class="err">Cannot get host filesystem geometry.</span>
<span class="err">Repair may fail if there is a sector size mismatch between</span>
<span class="err">the image and the host filesystem.</span>
<span class="err">Phase 1 - find and verify superblock...</span>
<span class="err">Cannot get host filesystem geometry.</span>
<span class="err">Repair may fail if there is a sector size mismatch between</span>
<span class="err">the image and the host filesystem.</span>
<span class="err">Phase 2 - using internal log</span>
<span class="err"> - zero log...</span>
<span class="c">ERROR: The filesystem has valuable metadata changes in a log which needs to</span>
<span class="err">be replayed. Mount the filesystem to replay the log, and unmount it before</span>
<span class="err">re-running xfs_repair. If you are unable to mount the filesystem, then use</span>
<span class="err">the -L option to destroy the log and attempt a repair.</span>
<span class="err">Note that destroying the log may cause corruption -- please attempt a mount</span>
<span class="err">of the filesystem before doing this.</span>
</pre></div>
<p>Actually, we shouldn't really try to do this directly on an image file. Better to do this on a loopback device.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="w"></span>
<span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">img</span><span class="w"> </span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">metadata</span><span class="w"> </span><span class="n">lost</span><span class="o">+</span><span class="k">found</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">losetup</span><span class="w"> </span><span class="c1">--show --find /mnt/server6-home.img</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop0</span><span class="w"></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_repair</span><span class="w"> </span><span class="o">-</span><span class="n">L</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop0</span><span class="w"></span>
<span class="n">Cannot</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="k">host</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">geometry</span><span class="p">.</span><span class="w"></span>
<span class="n">Repair</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">fail</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">there</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">sector</span><span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="n">mismatch</span><span class="w"> </span><span class="ow">between</span><span class="w"></span>
<span class="n">the</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">host</span><span class="w"> </span><span class="n">filesystem</span><span class="p">.</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">find</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">verify</span><span class="w"> </span><span class="n">superblock</span><span class="p">...</span><span class="w"></span>
<span class="o">[</span><span class="n">snipped</span><span class="o">]</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">7</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">verify</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">correct</span><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="n">counts</span><span class="p">...</span><span class="w"></span>
<span class="n">Note</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">quota</span><span class="w"> </span><span class="n">info</span><span class="w"> </span><span class="n">will</span><span class="w"> </span><span class="n">be</span><span class="w"> </span><span class="n">regenerated</span><span class="w"> </span><span class="k">on</span><span class="w"> </span><span class="k">next</span><span class="w"> </span><span class="n">quota</span><span class="w"> </span><span class="n">mount</span><span class="p">.</span><span class="w"></span>
<span class="n">done</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="o">/</span><span class="n">loopy</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop0</span><span class="w"> </span><span class="o">/</span><span class="n">loopy</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="n">loopy</span><span class="w"></span>
<span class="s1">''</span><span class="err">$</span><span class="s1">'\003''kovu'</span><span class="w"> </span><span class="s1">'Gb0'</span><span class="err">$</span><span class="s1">'\016''n'</span><span class="err">$</span><span class="s1">'\021''k'</span><span class="err">$</span><span class="s1">'\033'</span><span class="w"> </span><span class="s1">'bcCa-D'</span><span class="err">$</span><span class="s1">'\005\023''C'</span><span class="err">$</span><span class="s1">'\032''m'</span><span class="w"> </span><span class="s1">'ryN'</span><span class="err">$</span><span class="s1">'\004''xVX'</span><span class="err">$</span><span class="s1">'\024'</span><span class="w"></span>
<span class="s1">''</span><span class="err">$</span><span class="s1">'\a''hat5'</span><span class="w"> </span><span class="s1">'GpzR_Ksp'</span><span class="err">$</span><span class="s1">'\001\264''J0E'</span><span class="w"> </span><span class="s1">'h'</span><span class="err">$</span><span class="s1">'\t''rro\'</span><span class="w"> </span><span class="n">user11</span><span class="w"></span>
<span class="s1">''</span><span class="err">$</span><span class="s1">'\016''cpal'</span><span class="w"> </span><span class="s1">'Hwp-hBh'</span><span class="err">$</span><span class="s1">'\b''?;% '</span><span class="w"> </span><span class="s1">'j'</span><span class="err">$</span><span class="s1">'\003''pajK'</span><span class="w"> </span><span class="s1">'y7jG8CU4bonN2aujzfJamyN3xY'</span><span class="err">$</span><span class="s1">'\a\v\034''cs'</span><span class="w"></span>
<span class="s1">'-Xqcr'</span><span class="err">$</span><span class="s1">'\016''2o'</span><span class="err">$</span><span class="s1">'\005''k'</span><span class="w"> </span><span class="s1">'N6'</span><span class="err">$</span><span class="s1">'\001\343''vdA'</span><span class="w"> </span><span class="s1">'mF88U1Y-'</span><span class="err">$</span><span class="s1">'\016\033''Hs,'</span><span class="w"> </span><span class="s1">'yv8UsY'</span><span class="err">$</span><span class="s1">'\002\036''#R3'</span><span class="w"></span>
<span class="s1">'76q67pR'</span><span class="err">$</span><span class="s1">'\002''>]YT'</span><span class="w"> </span><span class="s1">'UROC0Bh9c'</span><span class="err">$</span><span class="s1">'\001''B'</span><span class="err">$</span><span class="s1">'\027'':'</span><span class="err">$</span><span class="s1">'\177'</span><span class="w"> </span><span class="s1">'n'</span><span class="err">$</span><span class="s1">'\001''tesb'</span><span class="w"></span>
<span class="s1">'8n-B'</span><span class="err">$</span><span class="s1">'\001''CItv'</span><span class="w"> </span><span class="s1">'WP4c'</span><span class="err">$</span><span class="s1">'\016''~Ia'</span><span class="err">$</span><span class="s1">'\037'</span><span class="w"> </span><span class="s1">'qsw77'</span><span class="err">$</span><span class="s1">'\016''D7x@'</span><span class="w"></span>
</pre></div>
<p>This is just a copy of the metadata, but doesn't look so good. I expected to at least see file names.
Let's try to copy the partition itself and work from that copy. That way, we can use destructive operations and we can ease up on slamming a failing disk with all this activity.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">dd</span><span class="w"> </span><span class="k">if</span><span class="o">=/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"> </span><span class="k">of</span><span class="o">=/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="o">-</span><span class="nc">real</span><span class="p">.</span><span class="n">img</span><span class="w"> </span><span class="n">bs</span><span class="o">=</span><span class="mi">128</span><span class="n">k</span><span class="w"></span>
<span class="nl">dd</span><span class="p">:</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">reading</span><span class="w"> </span><span class="s1">'/dev/centos/home'</span><span class="err">:</span><span class="w"> </span><span class="k">Input</span><span class="o">/</span><span class="k">output</span><span class="w"> </span><span class="n">error</span><span class="w"></span>
<span class="mi">11408</span><span class="o">+</span><span class="mi">0</span><span class="w"> </span><span class="n">records</span><span class="w"> </span><span class="ow">in</span><span class="w"></span>
<span class="mi">11408</span><span class="o">+</span><span class="mi">0</span><span class="w"> </span><span class="n">records</span><span class="w"> </span><span class="k">out</span><span class="w"></span>
<span class="mi">1495269376</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="p">(</span><span class="mf">1.5</span><span class="w"> </span><span class="n">GB</span><span class="p">,</span><span class="w"> </span><span class="mf">1.4</span><span class="w"> </span><span class="n">GiB</span><span class="p">)</span><span class="w"> </span><span class="n">copied</span><span class="p">,</span><span class="w"> </span><span class="mf">78.6088</span><span class="w"> </span><span class="n">s</span><span class="p">,</span><span class="w"> </span><span class="mf">19.0</span><span class="w"> </span><span class="n">MB</span><span class="o">/</span><span class="n">s</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="n">mnt</span><span class="w"></span>
<span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="o">-</span><span class="nc">real</span><span class="p">.</span><span class="n">img</span><span class="w"> </span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">img</span><span class="w"> </span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="p">.</span><span class="n">metadata</span><span class="w"> </span><span class="n">lost</span><span class="o">+</span><span class="k">found</span><span class="w"></span>
</pre></div>
<p>Nope. I/O errors. Let's try ddrescue.</p>
<p>We need to make 2 passes: one pass to read just the good blocks (those without read errors), and a second pass to try to read the bad the blocks three times before giving up:</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ddrescue</span><span class="w"> </span><span class="o">-</span><span class="n">f</span><span class="w"> </span><span class="o">-</span><span class="n">n</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="o">-</span><span class="n">rescue</span><span class="p">.</span><span class="n">img</span><span class="w"> </span><span class="n">rescue</span><span class="p">.</span><span class="nf">log</span><span class="w"></span>
<span class="n">GNU</span><span class="w"> </span><span class="n">ddrescue</span><span class="w"> </span><span class="mf">1.25</span><span class="w"></span>
<span class="n">Press</span><span class="w"> </span><span class="n">Ctrl</span><span class="o">-</span><span class="n">C</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">interrupt</span><span class="w"></span>
<span class="w"> </span><span class="nl">ipos</span><span class="p">:</span><span class="w"> </span><span class="mi">1495</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="w"> </span><span class="n">non</span><span class="o">-</span><span class="nl">trimmed</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="k">current</span><span class="w"> </span><span class="nl">rate</span><span class="p">:</span><span class="w"> </span><span class="mi">30720</span><span class="w"> </span><span class="n">B</span><span class="o">/</span><span class="n">s</span><span class="w"></span>
<span class="w"> </span><span class="nl">opos</span><span class="p">:</span><span class="w"> </span><span class="mi">1495</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="w"> </span><span class="n">non</span><span class="o">-</span><span class="nl">scraped</span><span class="p">:</span><span class="w"> </span><span class="mi">3072</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">average</span><span class="w"> </span><span class="nl">rate</span><span class="p">:</span><span class="w"> </span><span class="mi">17486</span><span class="w"> </span><span class="n">kB</span><span class="o">/</span><span class="n">s</span><span class="w"></span>
<span class="n">non</span><span class="o">-</span><span class="nl">tried</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">bad</span><span class="o">-</span><span class="nl">sector</span><span class="p">:</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="nl">rate</span><span class="p">:</span><span class="w"> </span><span class="mi">256</span><span class="w"> </span><span class="n">B</span><span class="o">/</span><span class="n">s</span><span class="w"></span>
<span class="w"> </span><span class="nl">rescued</span><span class="p">:</span><span class="w"> </span><span class="mi">96770</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="w"> </span><span class="n">bad</span><span class="w"> </span><span class="nl">areas</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="nc">time</span><span class="err">:</span><span class="w"> </span><span class="mi">1</span><span class="n">h</span><span class="w"> </span><span class="mi">32</span><span class="n">m</span><span class="w"> </span><span class="mi">14</span><span class="n">s</span><span class="w"></span>
<span class="n">pct</span><span class="w"> </span><span class="nl">rescued</span><span class="p">:</span><span class="w"> </span><span class="mf">99.99</span><span class="o">%</span><span class="p">,</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="nl">errors</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="n">remaining</span><span class="w"> </span><span class="nc">time</span><span class="err">:</span><span class="w"> </span><span class="mi">1</span><span class="n">s</span><span class="w"></span>
<span class="w"> </span><span class="nc">time</span><span class="w"> </span><span class="n">since</span><span class="w"> </span><span class="k">last</span><span class="w"> </span><span class="n">successful</span><span class="w"> </span><span class="k">read</span><span class="err">:</span><span class="w"> </span><span class="mi">0</span><span class="n">s</span><span class="w"></span>
<span class="n">Finished</span><span class="w"> </span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ddrescue</span><span class="w"> </span><span class="o">-</span><span class="n">d</span><span class="w"> </span><span class="o">-</span><span class="n">f</span><span class="w"> </span><span class="o">-</span><span class="n">r3</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">centos</span><span class="o">/</span><span class="n">home</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">home</span><span class="o">-</span><span class="n">rescue</span><span class="p">.</span><span class="n">img</span><span class="w"> </span><span class="n">rescue</span><span class="p">.</span><span class="nf">log</span><span class="w"></span>
<span class="n">GNU</span><span class="w"> </span><span class="n">ddrescue</span><span class="w"> </span><span class="mf">1.25</span><span class="w"></span>
<span class="n">Press</span><span class="w"> </span><span class="n">Ctrl</span><span class="o">-</span><span class="n">C</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">interrupt</span><span class="w"></span>
<span class="n">Initial</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="p">(</span><span class="k">read</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">mapfile</span><span class="p">)</span><span class="w"></span>
<span class="nl">rescued</span><span class="p">:</span><span class="w"> </span><span class="mi">96770</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="w"> </span><span class="nl">tried</span><span class="p">:</span><span class="w"> </span><span class="mi">4096</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">bad</span><span class="o">-</span><span class="nl">sector</span><span class="p">:</span><span class="w"> </span><span class="mi">1024</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">bad</span><span class="w"> </span><span class="nl">areas</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="k">Current</span><span class="w"> </span><span class="n">status</span><span class="w"></span>
<span class="w"> </span><span class="nl">ipos</span><span class="p">:</span><span class="w"> </span><span class="mi">1495</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="w"> </span><span class="n">non</span><span class="o">-</span><span class="nl">trimmed</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="k">current</span><span class="w"> </span><span class="nl">rate</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">B</span><span class="o">/</span><span class="n">s</span><span class="w"></span>
<span class="w"> </span><span class="nl">opos</span><span class="p">:</span><span class="w"> </span><span class="mi">1495</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="w"> </span><span class="n">non</span><span class="o">-</span><span class="nl">scraped</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">average</span><span class="w"> </span><span class="nl">rate</span><span class="p">:</span><span class="w"> </span><span class="mi">398</span><span class="w"> </span><span class="n">B</span><span class="o">/</span><span class="n">s</span><span class="w"></span>
<span class="n">non</span><span class="o">-</span><span class="nl">tried</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">bad</span><span class="o">-</span><span class="nl">sector</span><span class="p">:</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="n">B</span><span class="p">,</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="nl">rate</span><span class="p">:</span><span class="w"> </span><span class="mi">170</span><span class="w"> </span><span class="n">B</span><span class="o">/</span><span class="n">s</span><span class="w"></span>
<span class="w"> </span><span class="nl">rescued</span><span class="p">:</span><span class="w"> </span><span class="mi">96770</span><span class="w"> </span><span class="n">MB</span><span class="p">,</span><span class="w"> </span><span class="n">bad</span><span class="w"> </span><span class="nl">areas</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="nc">time</span><span class="err">:</span><span class="w"> </span><span class="mi">9</span><span class="n">s</span><span class="w"></span>
<span class="n">pct</span><span class="w"> </span><span class="nl">rescued</span><span class="p">:</span><span class="w"> </span><span class="mf">99.99</span><span class="o">%</span><span class="p">,</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="nl">errors</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="p">,</span><span class="w"> </span><span class="n">remaining</span><span class="w"> </span><span class="nc">time</span><span class="err">:</span><span class="w"> </span><span class="n">n</span><span class="o">/</span><span class="n">a</span><span class="w"></span>
<span class="w"> </span><span class="nc">time</span><span class="w"> </span><span class="n">since</span><span class="w"> </span><span class="k">last</span><span class="w"> </span><span class="n">successful</span><span class="w"> </span><span class="k">read</span><span class="err">:</span><span class="w"> </span><span class="mi">3</span><span class="n">s</span><span class="w"></span>
<span class="n">Finished</span><span class="w"> </span>
</pre></div>
<p>Now, let's try to grab the metadata from this copied partition image.
Mount the image, and then dump/restore the metadata to yet <em>another</em> image.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">losetup</span><span class="w"> </span><span class="c1">--find --show /mnt/server6-home-rescue.img</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop1</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_metadump</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop1</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">rescued</span><span class="o">-</span><span class="n">img</span><span class="p">.</span><span class="n">metadump</span><span class="w"></span>
<span class="nl">xfs_metadump</span><span class="p">:</span><span class="w"> </span><span class="nl">Warning</span><span class="p">:</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">recovery</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">an</span><span class="w"> </span><span class="n">obfuscated</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="n">leak</span><span class="w"> </span><span class="n">unobfuscated</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="ow">and</span><span class="o">/</span><span class="ow">or</span><span class="w"> </span><span class="n">cause</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="n">corruption</span><span class="p">.</span><span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="n">possible</span><span class="p">,</span><span class="w"> </span><span class="n">please</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">clean</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="p">,</span><span class="w"> </span><span class="ow">or</span><span class="w"> </span><span class="n">disable</span><span class="w"> </span><span class="n">obfuscation</span><span class="p">.</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_mdrestore</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">rescued</span><span class="o">-</span><span class="n">img</span><span class="p">.</span><span class="n">metadump</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">rescued</span><span class="o">-</span><span class="n">metadata</span><span class="p">.</span><span class="n">img</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_repair</span><span class="w"> </span><span class="o">/</span><span class="n">mnt</span><span class="o">/</span><span class="n">server6</span><span class="o">-</span><span class="n">rescued</span><span class="o">-</span><span class="n">metadata</span><span class="p">.</span><span class="n">img</span><span class="w"></span>
<span class="n">Cannot</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="k">host</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">geometry</span><span class="p">.</span><span class="w"></span>
<span class="n">Repair</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">fail</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">there</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">sector</span><span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="n">mismatch</span><span class="w"> </span><span class="ow">between</span><span class="w"></span>
<span class="n">the</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">host</span><span class="w"> </span><span class="n">filesystem</span><span class="p">.</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">find</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">verify</span><span class="w"> </span><span class="n">superblock</span><span class="p">...</span><span class="w"></span>
<span class="n">Cannot</span><span class="w"> </span><span class="k">get</span><span class="w"> </span><span class="k">host</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">geometry</span><span class="p">.</span><span class="w"></span>
<span class="n">Repair</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">fail</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">there</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">sector</span><span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="n">mismatch</span><span class="w"> </span><span class="ow">between</span><span class="w"></span>
<span class="n">the</span><span class="w"> </span><span class="nc">image</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">host</span><span class="w"> </span><span class="n">filesystem</span><span class="p">.</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">internal</span><span class="w"> </span><span class="nf">log</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="nf">log</span><span class="p">...</span><span class="w"></span>
<span class="nl">ERROR</span><span class="p">:</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">has</span><span class="w"> </span><span class="n">valuable</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="n">changes</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">which</span><span class="w"> </span><span class="n">needs</span><span class="w"> </span><span class="k">to</span><span class="w"></span>
<span class="n">be</span><span class="w"> </span><span class="n">replayed</span><span class="p">.</span><span class="w"> </span><span class="n">Mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">replay</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="p">,</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">unmount</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">before</span><span class="w"></span>
<span class="n">re</span><span class="o">-</span><span class="n">running</span><span class="w"> </span><span class="n">xfs_repair</span><span class="p">.</span><span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="k">are</span><span class="w"> </span><span class="n">unable</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="p">,</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="k">use</span><span class="w"></span>
<span class="n">the</span><span class="w"> </span><span class="o">-</span><span class="n">L</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">destroy</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">attempt</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">repair</span><span class="p">.</span><span class="w"></span>
<span class="n">Note</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">destroying</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">cause</span><span class="w"> </span><span class="n">corruption</span><span class="w"> </span><span class="c1">-- please attempt a mount</span>
<span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">before</span><span class="w"> </span><span class="n">doing</span><span class="w"> </span><span class="n">this</span><span class="p">.</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">losetup</span><span class="w"> </span><span class="c1">--show --find /mnt/server6-rescued-metadata.img</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop2</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">xfs_repair</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop2</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">find</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">verify</span><span class="w"> </span><span class="n">superblock</span><span class="p">...</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">internal</span><span class="w"> </span><span class="nf">log</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="nf">log</span><span class="p">...</span><span class="w"></span>
<span class="nl">ERROR</span><span class="p">:</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">has</span><span class="w"> </span><span class="n">valuable</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="n">changes</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">which</span><span class="w"> </span><span class="n">needs</span><span class="w"> </span><span class="k">to</span><span class="w"></span>
<span class="n">be</span><span class="w"> </span><span class="n">replayed</span><span class="p">.</span><span class="w"> </span><span class="n">Mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">replay</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="p">,</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">unmount</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">before</span><span class="w"></span>
<span class="n">re</span><span class="o">-</span><span class="n">running</span><span class="w"> </span><span class="n">xfs_repair</span><span class="p">.</span><span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="k">are</span><span class="w"> </span><span class="n">unable</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="p">,</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="k">use</span><span class="w"></span>
<span class="n">the</span><span class="w"> </span><span class="o">-</span><span class="n">L</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">destroy</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">attempt</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">repair</span><span class="p">.</span><span class="w"></span>
<span class="n">Note</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">destroying</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">cause</span><span class="w"> </span><span class="n">corruption</span><span class="w"> </span><span class="c1">-- please attempt a mount</span>
<span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">before</span><span class="w"> </span><span class="n">doing</span><span class="w"> </span><span class="n">this</span><span class="p">.</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="o">/</span><span class="n">mnt2</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop2</span><span class="w"> </span><span class="o">/</span><span class="n">mnt2</span><span class="w"></span>
<span class="nl">mount</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="nl">mnt2</span><span class="p">:</span><span class="w"> </span><span class="n">mount</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="k">system</span><span class="w"> </span><span class="k">call</span><span class="w"> </span><span class="nl">failed</span><span class="p">:</span><span class="w"> </span><span class="k">Structure</span><span class="w"> </span><span class="n">needs</span><span class="w"> </span><span class="n">cleaning</span><span class="p">.</span><span class="w"></span>
</pre></div>
<p>Okay. We expected that.<br>
Let's do the last-resort, fully destructive repair operation. Remember, we're still operating on the on the <em>metadata</em> image from the <em>copied</em> partition image.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi /]# xfs_repair -L /dev/loop2</span>
<span class="err">Phase 1 - find and verify superblock...</span>
<span class="err">Phase 2 - using internal log</span>
<span class="err"> - zero log...</span>
<span class="c">ALERT: The filesystem has valuable metadata changes in a log which is being</span>
<span class="err">destroyed because the -L option was used.</span>
<span class="err"> - scan filesystem freespace and inode maps...</span>
<span class="err">sb_fdblocks 1723017, counted 1731714</span>
<span class="err"> - found root inode chunk</span>
<span class="err">Phase 3 - for each AG...</span>
<span class="err"> - scan and clear agi unlinked lists...</span>
<span class="err"> - process known inodes and perform inode discovery...</span>
<span class="err"> - agno = 0</span>
<span class="err">Metadata corruption detected at 0x51b650, xfs_inode block 0x2c9000/0x2000</span>
<span class="err">bad magic number 0x0 on inode 5840900</span>
<span class="err">bad version number 0x0 on inode 5840900</span>
<span class="err">bad next_unlinked 0x0 on inode 5840900</span>
<span class="err">bad magic number 0x0 on inode 5840901</span>
<span class="err">bad version number 0x0 on inode 5840901</span>
<span class="err">bad next_unlinked 0x0 on inode 5840901</span>
<span class="err">bad magic number 0x0 on inode 5840900, resetting magic number</span>
<span class="err">bad version number 0x0 on inode 5840900, resetting version number</span>
<span class="err">bad next_unlinked 0x0 on inode 5840900, resetting next_unlinked</span>
<span class="err">imap claims a free inode 5840900 is in use, correcting imap and clearing inode</span>
<span class="err">cleared inode 5840900</span>
<span class="err">bad magic number 0x0 on inode 5840901, resetting magic number</span>
<span class="err">bad version number 0x0 on inode 5840901, resetting version number</span>
<span class="err">bad next_unlinked 0x0 on inode 5840901, resetting next_unlinked</span>
<span class="err">imap claims a free inode 5840901 is in use, correcting imap and clearing inode</span>
<span class="err">cleared inode 5840901</span>
<span class="err"> - agno = 1</span>
<span class="err">Metadata corruption detected at 0x50cd48, xfs_dir3_block block 0x2d24dd0/0x1000</span>
<span class="err"> - agno = 2</span>
<span class="err"> - agno = 3</span>
<span class="err"> - process newly discovered inodes...</span>
<span class="err">Phase 4 - check for duplicate blocks...</span>
<span class="err"> - setting up duplicate extent list...</span>
<span class="err"> - check for inodes claiming duplicate blocks...</span>
<span class="err"> - agno = 0</span>
<span class="err">entry "yW5knH-uj-UcvFmnM5CXvwZHrx?=0" at block 1238 offset 2992 in directory inode 13508451 references free inode 5840901</span>
<span class="err"> clearing inode number in entry at offset 2992...</span>
<span class="err">entry "Ja_rLm8K4VNoEIN8CR3ujtnulyRarKI1kIgwud6m 15t" at block 0 offset 344 in directory inode 66696251 references free inode 5840900</span>
<span class="err"> clearing inode number in entry at offset 344...</span>
<span class="err"> - agno = 1</span>
<span class="err">Metadata corruption detected at 0x50cd48, xfs_dir3_block block 0x2d24dd0/0x1000</span>
<span class="err"> - agno = 2</span>
<span class="err"> - agno = 3</span>
<span class="err">Phase 5 - rebuild AG headers and trees...</span>
<span class="err"> - reset superblock...</span>
<span class="err">Phase 6 - check inode connectivity...</span>
<span class="err"> - resetting contents of realtime bitmap and summary inodes</span>
<span class="err"> - traversing filesystem ...</span>
<span class="err">rebuilding directory inode 13508451</span>
<span class="err">bad hash table for directory inode 66696251 (no data entry): rebuilding</span>
<span class="err">rebuilding directory inode 66696251</span>
<span class="err">Metadata corruption detected at 0x50cd48, xfs_dir3_block block 0x2d24dd0/0x1000</span>
<span class="err">bad hash table for directory inode 134370432 (hash value mismatch): rebuilding</span>
<span class="err">rebuilding directory inode 134370432</span>
<span class="err">...</span>
</pre></div>
<p>Survey says....<em>hoooonk</em></p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi /]# mount /dev/loop2 /mnt2</span>
<span class="err">[root@alarmpi /]# ls /mnt2</span>
<span class="err">''$'\003''kovu' 'Gb0'$'\016''n'$'\021''k'$'\033' 'bcCa-D'$'\005\023''C'$'\032''m' 'ryN'$'\004''xVX'$'\024'</span>
<span class="err">''$'\a''hat5' 'GpzR_Ksp'$'\001\264''J0E' 'h'$'\t''rro\' user11</span>
<span class="err">''$'\016''cpal' 'Hwp-hBh'$'\b''?;% ' 'j'$'\003''pajK' 'y7jG8CU4bonN2aujzfJamyN3xY'$'\a\v\034''cs'</span>
<span class="err">'-Xqcr'$'\016''2o'$'\005''k' 'N6'$'\001\343''vdA' 'mF88U1Y-'$'\016\033''Hs,' 'yv8UsY'$'\002\036''#R3'</span>
<span class="err">'76q67pR'$'\002''>]YT' 'UROC0Bh9c'$'\001''B'$'\027'':'$'\177' 'n'$'\001''tesb'</span>
<span class="err">'8n-B'$'\001''CItv' 'WP4c'$'\016''~Ia'$'\037' 'qsw77'$'\016''D7x@'</span>
</pre></div>
<p>Fine. Let's just try the operation directly on the copied partition image created from ddrescue.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">alarm@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">losetup</span><span class="w"> </span><span class="c1">--find --show /mnt/server6-home-rescue.img</span>
<span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop3</span><span class="w"></span>
<span class="o">[</span><span class="n">alarm@alarmpi /</span><span class="o">]</span><span class="err">$</span><span class="w"> </span><span class="n">sudo</span><span class="w"> </span><span class="n">xfs_repair</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop3</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">find</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">verify</span><span class="w"> </span><span class="n">superblock</span><span class="p">...</span><span class="w"></span>
<span class="n">Phase</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">internal</span><span class="w"> </span><span class="nf">log</span><span class="w"></span>
<span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">zero</span><span class="w"> </span><span class="nf">log</span><span class="p">...</span><span class="w"></span>
<span class="nl">ERROR</span><span class="p">:</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="n">has</span><span class="w"> </span><span class="n">valuable</span><span class="w"> </span><span class="n">metadata</span><span class="w"> </span><span class="n">changes</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">which</span><span class="w"> </span><span class="n">needs</span><span class="w"> </span><span class="k">to</span><span class="w"></span>
<span class="n">be</span><span class="w"> </span><span class="n">replayed</span><span class="p">.</span><span class="w"> </span><span class="n">Mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">replay</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="p">,</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">unmount</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">before</span><span class="w"></span>
<span class="n">re</span><span class="o">-</span><span class="n">running</span><span class="w"> </span><span class="n">xfs_repair</span><span class="p">.</span><span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="n">you</span><span class="w"> </span><span class="k">are</span><span class="w"> </span><span class="n">unable</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="p">,</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="k">use</span><span class="w"></span>
<span class="n">the</span><span class="w"> </span><span class="o">-</span><span class="n">L</span><span class="w"> </span><span class="k">option</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">destroy</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">attempt</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">repair</span><span class="p">.</span><span class="w"></span>
<span class="n">Note</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">destroying</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="n">cause</span><span class="w"> </span><span class="n">corruption</span><span class="w"> </span><span class="c1">-- please attempt a mount</span>
<span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">filesystem</span><span class="w"> </span><span class="k">before</span><span class="w"> </span><span class="n">doing</span><span class="w"> </span><span class="n">this</span><span class="p">.</span><span class="w"></span>
<span class="o">[</span><span class="n">alarm@alarmpi /</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mkdir</span><span class="w"> </span><span class="o">/</span><span class="n">mnt3</span><span class="w"> </span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop3</span><span class="w"> </span><span class="o">/</span><span class="n">mnt3</span><span class="w"></span>
<span class="nl">mount</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="nl">mnt3</span><span class="p">:</span><span class="w"> </span><span class="n">mount</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="k">system</span><span class="w"> </span><span class="k">call</span><span class="w"> </span><span class="nl">failed</span><span class="p">:</span><span class="w"> </span><span class="k">Structure</span><span class="w"> </span><span class="n">needs</span><span class="w"> </span><span class="n">cleaning</span><span class="p">.</span><span class="w"></span>
</pre></div>
<p>Okay. As expected.<br>
Let's go straight to -L.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# xfs_repair -L /dev/loop3</span>
<span class="err">Phase 1 - find and verify superblock...</span>
<span class="err">Phase 2 - using internal log</span>
<span class="err"> - zero log...</span>
<span class="c">ALERT: The filesystem has valuable metadata changes in a log which is being</span>
<span class="err">destroyed because the -L option was used.</span>
<span class="err"> - scan filesystem freespace and inode maps...</span>
<span class="err">sb_fdblocks 1723017, counted 1731714</span>
<span class="err"> - found root inode chunk</span>
<span class="err">Phase 3 - for each AG...</span>
<span class="err"> - scan and clear agi unlinked lists...</span>
<span class="err"> - process known inodes and perform inode discovery...</span>
<span class="err"> - agno = 0</span>
<span class="err">Metadata corruption detected at 0x553650, xfs_inode block 0x2c9000/0x2000</span>
<span class="err">bad magic number 0x0 on inode 5840900</span>
<span class="err">bad version number 0x0 on inode 5840900</span>
<span class="err">bad next_unlinked 0x0 on inode 5840900</span>
<span class="err">bad magic number 0x0 on inode 5840901</span>
<span class="err">bad version number 0x0 on inode 5840901</span>
<span class="err">bad next_unlinked 0x0 on inode 5840901</span>
<span class="err">bad magic number 0x0 on inode 5840900, resetting magic number</span>
<span class="err">bad version number 0x0 on inode 5840900, resetting version number</span>
<span class="err">bad next_unlinked 0x0 on inode 5840900, resetting next_unlinked</span>
<span class="err">imap claims a free inode 5840900 is in use, correcting imap and clearing inode</span>
<span class="err">cleared inode 5840900</span>
<span class="err">bad magic number 0x0 on inode 5840901, resetting magic number</span>
<span class="err">bad version number 0x0 on inode 5840901, resetting version number</span>
<span class="err">bad next_unlinked 0x0 on inode 5840901, resetting next_unlinked</span>
<span class="err">imap claims a free inode 5840901 is in use, correcting imap and clearing inode</span>
<span class="err">cleared inode 5840901</span>
<span class="err"> - agno = 1</span>
<span class="err"> - agno = 2</span>
<span class="err"> - agno = 3</span>
<span class="err"> - process newly discovered inodes...</span>
<span class="err">Phase 4 - check for duplicate blocks...</span>
<span class="err"> - setting up duplicate extent list...</span>
<span class="err"> - check for inodes claiming duplicate blocks...</span>
<span class="err"> - agno = 0</span>
<span class="err">entry "sess_57usp4fkeg1icc0pbkmju3nn34" at block 1238 offset 2992 in directory inode 13508451 references free inode 5840901</span>
<span class="err"> clearing inode number in entry at offset 2992...</span>
<span class="err">entry "zend_cache---internal-metadatas---artist_skin" at block 0 offset 344 in directory inode 66696251 references free inode 5840900</span>
<span class="err"> clearing inode number in entry at offset 344...</span>
<span class="err"> - agno = 1</span>
<span class="err"> - agno = 2</span>
<span class="err"> - agno = 3</span>
<span class="err">Phase 5 - rebuild AG headers and trees...</span>
<span class="err"> - reset superblock...</span>
<span class="err">Phase 6 - check inode connectivity...</span>
<span class="err"> - resetting contents of realtime bitmap and summary inodes</span>
<span class="err"> - traversing filesystem ...</span>
<span class="err">rebuilding directory inode 13508451</span>
<span class="err">bad hash table for directory inode 66696251 (no data entry): rebuilding</span>
<span class="err">rebuilding directory inode 66696251</span>
<span class="err"> - traversal finished ...</span>
<span class="err"> - moving disconnected inodes to lost+found ...</span>
<span class="err">Phase 7 - verify and correct link counts...</span>
<span class="err">Note - quota info will be regenerated on next quota mount.</span>
<span class="err">done</span>
</pre></div>
<p>Let's give it a try...</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">mount</span><span class="w"> </span><span class="o">/</span><span class="n">dev</span><span class="o">/</span><span class="n">loop3</span><span class="w"> </span><span class="o">/</span><span class="n">mnt3</span><span class="w"></span>
<span class="o">[</span><span class="n">root@alarmpi ~</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">ls</span><span class="w"> </span><span class="o">/</span><span class="n">mnt3</span><span class="w"></span>
<span class="mi">0</span><span class="n">_README_BEFORE_DELETING_VIRTFS</span><span class="w"> </span><span class="n">aquota</span><span class="p">.</span><span class="k">user</span><span class="w"> </span><span class="n">user1</span><span class="w"> </span><span class="n">user2</span><span class="w"> </span><span class="n">user3</span><span class="w"> </span><span class="n">user4</span><span class="w"> </span><span class="n">quota</span><span class="p">.</span><span class="k">user</span><span class="w"> </span><span class="n">user5</span><span class="w"> </span><span class="n">user0</span><span class="w"> </span><span class="n">cPanelInstall</span><span class="w"> </span><span class="n">cpanelsolr</span><span class="w"> </span><span class="n">user10</span><span class="w"> </span><span class="n">user6</span><span class="w"> </span><span class="n">user7</span><span class="w"> </span><span class="n">user9</span><span class="w"> </span><span class="n">quota</span><span class="p">.</span><span class="k">group</span><span class="w"> </span><span class="n">virtfs</span><span class="w"></span>
</pre></div>
<p>WTF? Success?
I honestly did not expect that. </p>
<p>Some of the files themselves may be corrupted (probably from being in an inconsistent state when all this went down), but after poking around a bit, it appears that this does indeed contain sufficient data to make the recovery efforts worthwhile.</p>
<p>Copy the homedirs out of the mounted image and onto the external USB drive to upload later.</p>
<div class="highlight"><pre><span></span><span class="err">[root@alarmpi ~]# cd /mnt3</span>
<span class="err">[root@alarmpi mnt3]# for i in user1 user10 user3 user6 user7 user9 user4 user5 user0 user2; do echo $i; tar -cf /mnt/$i.tar $i; echo "$i done!"; done</span>
</pre></div>
<p>References:
https://serverfault.com/questions/777299/proper-way-to-deal-with-corrupt-xfs-filesystems
https://wiki.archlinux.org/index.php/Disk_cloning#Using_ddrescue
https://wiki.archlinux.org/index.php/XFS#Repair_XFS_Filesystem
https://jlk.fjfi.cvut.cz/arch/manpages/man/xfs_repair.8</p>
<p>Happily, did not have to do this one:
https://serverfault.com/a/834853</p>OwnCloud LDAP Authentication Using FreeIPA2020-02-29T00:00:00-05:002020-02-29T00:00:00-05:00Andrew Boringtag:andrewboring.com,2020-02-29:/content/owncloud-ldap-authentication-using-freeipa.html<p>The ownCloud software is a bit challenging to configure for LDAP-based authentication and authorization. The configuration screen uses Javascript to save the settings without clicking a "save" button, and some settings don't quite go away just because you unchecked the parent box. If you configure a setting with multiple attributes …</p><p>The ownCloud software is a bit challenging to configure for LDAP-based authentication and authorization. The configuration screen uses Javascript to save the settings without clicking a "save" button, and some settings don't quite go away just because you unchecked the parent box. If you configure a setting with multiple attributes and then want to disable that setting, it's best to disable any attributes (if possible) when disabling the setting itself. I don't have a ready example of this right now, unfortunately.</p>
<p>In this scenario, I want to require an email address for the username, and restrict it to the members of the group "owncloud".</p>
<p>The object classes selectors may not be absolutely necessary, but they may reduce search time if you have a large directory with many objects.</p>
<div class="highlight"><pre><span></span><span class="n">Server</span><span class="p">:</span>
<span class="o">-</span> <span class="n">Server</span><span class="p">:</span> <span class="n">ipa</span><span class="p">.</span><span class="n">sys</span><span class="p">.</span><span class="n">example</span><span class="p">.</span><span class="n">com</span>
<span class="o">-</span> <span class="n">Bind</span> <span class="n">DN</span><span class="p">:</span> <span class="n">uid</span><span class="o">=</span><span class="n">owncloud</span><span class="o">-</span><span class="k">system</span><span class="p">,</span><span class="n">cn</span><span class="o">=</span><span class="n">users</span><span class="p">,</span><span class="n">cn</span><span class="o">=</span><span class="n">accounts</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">sys</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">example</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">com</span>
<span class="o">-</span> <span class="n">Password</span><span class="p">:</span>
<span class="o">-</span> <span class="n">Base</span> <span class="n">DN</span><span class="p">:</span> <span class="n">dc</span><span class="o">=</span><span class="n">sys</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">example</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">com</span>
<span class="n">Users</span><span class="p">:</span>
<span class="o">-</span> <span class="k">Only</span> <span class="n">these</span> <span class="k">object</span> <span class="n">classes</span><span class="p">:</span> <span class="n">posixaccount</span>
<span class="n">Login</span> <span class="n">Attributes</span><span class="p">:</span>
<span class="o">-</span> <span class="n">LDAP</span> <span class="o">/</span> <span class="n">AD</span> <span class="n">Email</span> <span class="n">Address</span>
<span class="n">Groups</span><span class="p">:</span>
<span class="o">-</span> <span class="k">Only</span> <span class="n">these</span> <span class="k">object</span> <span class="n">classes</span><span class="p">:</span> <span class="n">posixgroup</span>
<span class="o">-</span> <span class="k">Only</span> <span class="k">from</span> <span class="n">these</span> <span class="n">groups</span><span class="p">:</span> <span class="n">owncloud</span>
</pre></div>SuiteCRM LDAP Authentication Using FreeIPA2020-02-29T00:00:00-05:002020-02-29T00:00:00-05:00Andrew Boringtag:andrewboring.com,2020-02-29:/content/suitecrm-ldap-authentication-using-freeipa.html<p>SuiteCRM LDAP Authentication Using FreeIPA.</p>
<p>The biggest problem with SuiteCRM in LDAP configuration is that there is no "test connection" button to validate the configuration is correct before saving it. Your only testing option is to save the configuration, then open a new window (Incognito mode or a different browser …</p><p>SuiteCRM LDAP Authentication Using FreeIPA.</p>
<p>The biggest problem with SuiteCRM in LDAP configuration is that there is no "test connection" button to validate the configuration is correct before saving it. Your only testing option is to save the configuration, then open a new window (Incognito mode or a different browser entirely) to test the login while you make additional changes.</p>
<p>This can be an extensive trial-and-error process, especially if you lack expertise in x509 directory schemas. Most of the documentation and online forum questions relate to Active Directory or OpenLDAP, rather than FreeIPA.</p>
<p>Requirements:
SuiteCRM user authenticates using a user ID (uid).
Authorized User is a staff member (member of the Staff group).
Internal LDAP realm is "sys.example.com".</p>
<div class="highlight"><pre><span></span><span class="n">Enable</span> <span class="n">LDAP</span> <span class="n">Authentication</span><span class="p">:</span> <span class="k">checked</span>
<span class="o">-</span> <span class="n">Server</span><span class="p">:</span> <span class="n">ipa</span><span class="p">.</span><span class="n">sys</span><span class="p">.</span><span class="n">eriscape</span><span class="p">.</span><span class="n">com</span>
<span class="o">-</span> <span class="n">Port</span> <span class="nb">Number</span><span class="p">:</span>
<span class="o">-</span> <span class="k">User</span> <span class="n">DN</span><span class="p">:</span> <span class="n">cn</span><span class="o">=</span><span class="n">users</span><span class="p">,</span><span class="n">cn</span><span class="o">=</span><span class="n">accounts</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">sys</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">example</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">com</span>
<span class="k">User</span> <span class="n">Filter</span><span class="p">:</span>
<span class="o">-</span> <span class="n">Bind</span> <span class="n">Attribute</span><span class="p">:</span> <span class="n">dn</span>
<span class="o">-</span> <span class="n">Login</span> <span class="n">Attribute</span><span class="p">:</span> <span class="n">uid</span>
<span class="n">Authentication</span>
<span class="o">-</span> <span class="k">User</span> <span class="n">Name</span><span class="p">:</span> <span class="n">uid</span><span class="o">=</span><span class="n">crm</span><span class="o">-</span><span class="k">system</span><span class="p">,</span><span class="n">cn</span><span class="o">=</span><span class="n">users</span><span class="p">,</span><span class="n">cn</span><span class="o">=</span><span class="n">accounts</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">sys</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">example</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">com</span>
<span class="o">-</span> <span class="n">Password</span><span class="p">:</span> <span class="o">*******</span>
</pre></div>
<p>Get this working first. You must be able to login using a username before mucking with group membership.
The "crm-system" user is a general system account for accessing the FreeIPA directory, and this is expressed as a DN rather than the user@REALM format you might enter for Active Directory. I like to create a separate system user for each service authenticating against LDAP to aid in troubleshooting/log analysis.</p>
<p>There are two general schemas in use, RFC2307 and RFC2307bis, and the biggest difference (to us) is how it identifies group membership. I haven't poked at this extensively, but I typically use memberOf on most systems that support 2307bis. This may not be supported by SuiteCRM, or it may be that other attributes need to be adjusted to support memberOf. <a href="https://unofficialaciguide.com/2019/07/31/ldap-schemas-for-aci-administrators-rfc2307-vs-rfc2307bis/">I wrote about this for a different blog</a>, so I won't detail it here.</p>
<p>To authorize only specific group members, configure the Group Membership section (where "staff" is the name of the group you want to restrict):</p>
<div class="highlight"><pre><span></span><span class="k">Group</span> <span class="n">Membership</span>
<span class="o">-</span> <span class="k">Group</span> <span class="n">DN</span><span class="p">:</span> <span class="n">cn</span><span class="o">=</span><span class="n">groups</span><span class="p">,</span><span class="n">cn</span><span class="o">=</span><span class="n">accounts</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">sys</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">example</span><span class="p">,</span><span class="n">dc</span><span class="o">=</span><span class="n">com</span>
<span class="o">-</span> <span class="k">Group</span> <span class="n">Name</span><span class="p">:</span> <span class="n">cn</span><span class="o">=</span><span class="n">staff</span>
<span class="o">-</span> <span class="k">User</span> <span class="n">attribute</span><span class="p">:</span> <span class="n">uid</span>
<span class="o">-</span> <span class="k">Group</span> <span class="n">Attribute</span><span class="p">:</span> <span class="n">member</span>
<span class="o">-</span> <span class="k">With</span> <span class="k">User</span> <span class="n">DN</span><span class="p">:</span> <span class="k">checked</span>
<span class="n">Auto</span> <span class="k">Create</span> <span class="n">Users</span><span class="p">:</span> <span class="k">checked</span>
</pre></div>
<p>The "Auto Create Users" must be checked so that SuiteCRM can create the necessary local database entries in order to apply roles/policies. SuiteCRM does not support policy/role definitions in LDAP.</p>Boot Images for Arch Linux ARM2019-11-21T00:00:00-05:002019-11-21T00:00:00-05:00Andrew Boringtag:andrewboring.com,2019-11-21:/content/alarm-images.html<p>I've been working a bit with Raspberry Pi lately, both the 3B and 3B+ and soon, the Pi 4. Most of this was originally to support a client using the 3B to drive media displays and interactive kiosks. Since they were using Arch Linux ARM, I began using it, too …</p><p>I've been working a bit with Raspberry Pi lately, both the 3B and 3B+ and soon, the Pi 4. Most of this was originally to support a client using the 3B to drive media displays and interactive kiosks. Since they were using Arch Linux ARM, I began using it, too.</p>
<p>Arch Linux ARM (ALArm, for short) provides a compressed tarball of the root filesystem, and you're responsible for preparing the SD card with the partitions and filesystems and unpacking the tarball onto the SD card from another computer before you can boot it. There's no installer in the traditional sense.</p>
<p>This is fine for one-off usage, but if you're building a provisioning tool or install script, you probably want a clean image each time. You could simply duplicate your card with an SD Card duplicator, but most people don't have one of these. The other option is writing the SD Card to an image file and back again.</p>
<p>Except...the smallest conventional card you can bu y these days is 32GB. The 8GB/16GB cards are still available, but are older units and are more expensive per-GB. Keeping a 32GB image file on disk takes up considerable disk space, while writing the image <em>back</em> to SD card can easily take a good four hours to complete. So, each time my script errored out and I needed to make an adjustment and boot a clean image, it was a time-consuming process. And since this was ARM instead of Intel, I can't simply use Virtualbox. Using Qemu to emulate ARM is doable to a small extent, but there are significant differences in a running OS in an emulator versus having actual supported hardware.</p>
<h2>Solution</h2>
<p>The solution is prepared 8GB image files, built directly from the upstream Arch Linux ARM projects distribution files.</p>
<p>This is easy enough on Linux; but if you're using macOS, working with Arch Linux-based systems is challenging at best:</p>
<ul>
<li>Creating BIOS partitions on UEFI/GPT Macs is a pain, and even more so on raw image files.</li>
<li>No [official] macOS support for Ext4 file systems. Third-party / FUSE options only.</li>
<li>Arch uses extended attributes in the bsdtar program, which is not the same as BSD tar, apparently, and certainly not GNU tar. I still don't get this one, personally.</li>
<li>Setting up nullfs ("bind mounts" in the Linux world) using the loopback device is not well documented on macOS.</li>
</ul>
<p>The simplest solution I found to create images is to use Vagrant+Virtualbox to spin up an Arch Linux virtual machine (any other distro will work, too), create the image file, mount it via loop dev, and then download/unpack the tarball into the image file via the bind mounts. After than, you can unmount the image file, copy it off somewhere, and destroy the Vagrant box. <a href="https://disconnected.systems/blog/raspberry-pi-archlinuxarm-setup/">This blog post</a> has the details, as I just modified from his guide.</p>
<h2>Automation</h2>
<p>However, I wanted to update these images periodically and hate Hate HATE!!! repetitive, manual tasks. I know I would let this crap get way out of date if I a) wasn't getting paid to pay attention to it and b) if I can get along without doing the extra work. I'm just lazy that way.</p>
<p>Happily, Disconnected Systems <a href="https://disconnected.systems/blog/custom-rpi-image-with-github-travis/">solved this one</a>, too, and after a bit of tinkering, I now have a working source repo that allows me to a) build/test images locally using Vagrant and b) automatically build/test/deploy releases directly to Github using Travis CI.</p>
<p>At the moment, I still need to manually create a release in Github, and then push a commit to trigger a build in Travis. The next improvement might be to have a monitor script that checks the upstream distribution for changes, then automatically creates a new build and a release.</p>
<p>Current releases can be found here: https://github.com/andrewboring/alarm-images/releases
Source repo (to build yourself) can be found here: https://github.com/andrewboring/alarm-images</p>Let's Encrypt with Wordpress Multisite2019-10-03T00:00:00-04:002019-10-03T00:00:00-04:00Andrew Boringtag:andrewboring.com,2019-10-03:/content/wp-letsencrypt.html<p>With Wordpress Multisite on a single webroot (eg, /var/www/html), we can define multiple domain namespaces and point them at any internal directory structure we want. That is, we can easily configure Apache such that a request for the domain/path goes anywhere on the server:</p>
<p>example.com -> /var …</p><p>With Wordpress Multisite on a single webroot (eg, /var/www/html), we can define multiple domain namespaces and point them at any internal directory structure we want. That is, we can easily configure Apache such that a request for the domain/path goes anywhere on the server:</p>
<p>example.com -> /var/www/html<br>
example.com/path -> /var/www/html/path<br>
example.com/path2 -> /home/data/path2<br>
example2.com -> /var/www/html/example2<br>
subdomain.example2.com -> /var/www/subdomain-example2 </p>
<p>For letsencrypt, let's point all requests for /.well-known/acme-challenge validation - <em>irrespective of the domain</em> - to a single directory that certbot can write to. That means we do not have to run a separate validation for each domain renewal, and we don't have to stop/start the web server so Certbot can respond to the the challenge independently (and don't even <em>think</em> about futzing with TXT records for DNS validation).</p>
<p><strong>Assumptions:</strong>
Single-tenant host system (VPS, physical server, EC2 instance, etc, NOT cPanel shared hosting)
Apache web server<br>
ModAlias
ModSSL
Certbot</p>
<p>I used CentOS 7 and Apache on a single virtual machine instance for this.</p>
<p>In your webroot, create the validation directory. Certbot will write all challenge files here.</p>
<div class="highlight"><pre><span></span><span class="err"># mkdir -p /var/www/html/.well-known/acme-challenge</span>
</pre></div>
<p><strong>Default</strong> Apache Virtualhost Config for Wordpress Multi-site
Apache will read all .conf files in alphabetical order, and the very first Virtualhost configuration in that order will be the "default" vhost served for a hostname if none other matches. Name this file 00-default.conf or similar to ensure that it is the first one read by Apache (the number "0" comes before the letter "a" in computing) if you have other sites served by Apache.</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">root@s html</span><span class="o">]</span><span class="err">#</span><span class="w"> </span><span class="n">cat</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">httpd</span><span class="o">/</span><span class="n">conf</span><span class="p">.</span><span class="n">d</span><span class="o">/</span><span class="mi">00</span><span class="o">-</span><span class="k">default</span><span class="p">.</span><span class="n">conf</span><span class="w"></span>
<span class="err">#</span><span class="w"> </span><span class="n">This</span><span class="w"> </span><span class="n">rewrites</span><span class="w"> </span><span class="n">configures</span><span class="w"> </span><span class="ow">all</span><span class="w"> </span><span class="n">requests</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">single</span><span class="w"> </span><span class="n">challenge</span><span class="w"> </span><span class="n">root</span><span class="p">.</span><span class="w"></span>
<span class="n">AliasMatch</span><span class="w"> </span><span class="o">^/</span><span class="p">.</span><span class="n">well</span><span class="o">-</span><span class="n">known</span><span class="o">/</span><span class="n">acme</span><span class="o">-</span><span class="n">challenge</span><span class="o">/</span><span class="p">(.</span><span class="o">*</span><span class="p">)</span><span class="err">$</span><span class="w"> </span><span class="o">/</span><span class="nf">var</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">html</span><span class="o">/</span><span class="p">.</span><span class="n">well</span><span class="o">-</span><span class="n">known</span><span class="o">/</span><span class="n">acme</span><span class="o">-</span><span class="n">challenge</span><span class="o">/</span><span class="err">$</span><span class="mi">1</span><span class="w"></span>
<span class="n">SSLStrictSNIVHostCheck</span><span class="w"> </span><span class="k">off</span><span class="w"></span>
<span class="err">#</span><span class="w"> </span><span class="n">I</span><span class="w"> </span><span class="n">run</span><span class="w"> </span><span class="n">Wordpress</span><span class="w"> </span><span class="o">[</span><span class="n">grudgingly</span><span class="o">]</span><span class="w"> </span><span class="k">out</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">wp</span><span class="w"> </span><span class="n">directory</span><span class="p">.</span><span class="w"></span>
<span class="err">#</span><span class="w"> </span><span class="n">I</span><span class="w"> </span><span class="n">keep</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">few</span><span class="w"> </span><span class="k">host</span><span class="o">-</span><span class="k">specific</span><span class="w"> </span><span class="n">files</span><span class="w"> </span><span class="k">under</span><span class="w"> </span><span class="o">/</span><span class="nf">var</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">html</span><span class="p">.</span><span class="w"></span>
<span class="o"><</span><span class="n">VirtualHost</span><span class="w"> </span><span class="mf">198.51.100.57</span><span class="err">:</span><span class="mi">80</span><span class="o">></span><span class="w"></span>
<span class="w"> </span><span class="n">ServerName</span><span class="w"> </span><span class="n">example</span><span class="p">.</span><span class="n">com</span><span class="w"></span>
<span class="w"> </span><span class="n">DocumentRoot</span><span class="w"> </span><span class="o">/</span><span class="nf">var</span><span class="o">/</span><span class="n">www</span><span class="o">/</span><span class="n">html</span><span class="o">/</span><span class="n">wp</span><span class="w"></span>
<span class="w"> </span><span class="n">ServerAdmin</span><span class="w"> </span><span class="n">goaway</span><span class="nv">@example</span><span class="p">.</span><span class="n">com</span><span class="w"></span>
<span class="w"> </span><span class="n">ErrorLog</span><span class="w"> </span><span class="n">logs</span><span class="o">/</span><span class="n">example</span><span class="p">.</span><span class="n">com</span><span class="o">-</span><span class="n">error_log</span><span class="w"></span>
<span class="w"> </span><span class="n">LogLevel</span><span class="w"> </span><span class="n">alert</span><span class="w"> </span><span class="nl">rewrite</span><span class="p">:</span><span class="n">trace3</span><span class="w"></span>
<span class="w"> </span><span class="n">TransferLog</span><span class="w"> </span><span class="n">logs</span><span class="o">/</span><span class="n">example</span><span class="p">.</span><span class="n">com</span><span class="o">-</span><span class="n">access_log</span><span class="w"></span>
<span class="o"></</span><span class="n">VirtualHost</span><span class="o">></span><span class="w"></span>
<span class="o"><</span><span class="n">VirtualHost</span><span class="w"> </span><span class="mf">198.51.100.57</span><span class="err">:</span><span class="mi">443</span><span class="o">></span><span class="w"></span>
<span class="w"> </span><span class="n">DocumentRoot</span><span class="w"> </span><span class="ss">"/var/www/html/wp"</span><span class="w"></span>
<span class="w"> </span><span class="n">ServerName</span><span class="w"> </span><span class="n">example</span><span class="p">.</span><span class="nl">com</span><span class="p">:</span><span class="mi">443</span><span class="w"></span>
<span class="w"> </span><span class="n">ErrorLog</span><span class="w"> </span><span class="n">logs</span><span class="o">/</span><span class="n">example</span><span class="p">.</span><span class="n">com</span><span class="o">-</span><span class="n">ssl_error_log</span><span class="w"></span>
<span class="w"> </span><span class="n">TransferLog</span><span class="w"> </span><span class="n">logs</span><span class="o">/</span><span class="n">example</span><span class="p">.</span><span class="n">com</span><span class="o">-</span><span class="n">ssl_access_log</span><span class="w"></span>
<span class="w"> </span><span class="n">LogLevel</span><span class="w"> </span><span class="n">warn</span><span class="w"></span>
<span class="w"> </span><span class="n">SSLEngine</span><span class="w"> </span><span class="k">on</span><span class="w"></span>
<span class="w"> </span><span class="n">SSLProtocol</span><span class="w"> </span><span class="ow">all</span><span class="w"> </span><span class="o">-</span><span class="n">SSLv2</span><span class="w"> </span><span class="o">-</span><span class="n">SSLv3</span><span class="w"></span>
<span class="w"> </span><span class="n">SSLCipherSuite</span><span class="w"> </span><span class="nl">HIGH</span><span class="p">:</span><span class="mi">3</span><span class="nl">DES</span><span class="p">:</span><span class="err">!</span><span class="nl">aNULL</span><span class="p">:</span><span class="err">!</span><span class="nl">MD5</span><span class="p">:</span><span class="err">!</span><span class="nl">SEED</span><span class="p">:</span><span class="err">!</span><span class="n">IDEA</span><span class="w"></span>
<span class="w"> </span><span class="n">SSLCertificateFile</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">live</span><span class="o">/</span><span class="n">example</span><span class="p">.</span><span class="n">com</span><span class="o">/</span><span class="n">fullchain</span><span class="p">.</span><span class="n">pem</span><span class="w"></span>
<span class="w"> </span><span class="n">SSLCertificateKeyFile</span><span class="w"> </span><span class="o">/</span><span class="n">etc</span><span class="o">/</span><span class="n">letsencrypt</span><span class="o">/</span><span class="n">live</span><span class="o">/</span><span class="n">example</span><span class="p">.</span><span class="n">com</span><span class="o">/</span><span class="n">privkey</span><span class="p">.</span><span class="n">pem</span><span class="w"></span>
<span class="o"></</span><span class="n">VirtualHost</span><span class="o">></span><span class="w"> </span>
</pre></div>
<p>The AliasMatch directive will map the ACME validation path to each domain that Apache serves.</p>
<p>So as long as Apache or Wordpress is configured to respond against that domain name (eg, with the domain mapping entry already applied to a given Wordpress site), it will match all URL paths of the form ".well-known/acme-challenge" to whatever domain is being requested.</p>
<p>When it's time to run certbot, use something like this:</p>
<div class="highlight"><pre><span></span><span class="err">certbot certonly --webroot -w /var/www/html --cert-name example.com \ </span>
<span class="err"> -d example.com -d www.example.com \ </span>
<span class="err"> -d example2.com -d www.example2.com \ </span>
<span class="err"> -d example3.com -d www.example3.com --dry-run</span>
</pre></div>
<p>I spread this across four lines for readability, but this is entered as one line.</p>
<ul>
<li>The certonly directive tells certbot not to mess with our web server config.</li>
<li>--webroot means use the webroot challenge, not DN</li>
<li>-w tells us the main webroot where it can find the /.well-known directory to write the challenge files.</li>
<li>--cert-name is the primary cert for the wordpress multisite. Here, it's example.com.</li>
<li>-d [domain] adds the domain(s) to the certificate.</li>
<li>--dry-run uses EFF's test servers, rather than slamming their production servers with failed requests causing them to rate-limit you to prevent a denial-of-service attack.</li>
</ul>New Site2019-09-23T00:00:00-04:002019-09-23T00:00:00-04:00Andrew Boringtag:andrewboring.com,2019-09-23:/content/new-site.html<p>For about two years, I've had a "new website" with nothing but the word "blargh" on it and some ancient crap I wrote years ago.</p>
<p>More recently, I needed to assemble a portfolio (both technical and musical) for types of work that I wanted to be doing, and this seemed …</p><p>For about two years, I've had a "new website" with nothing but the word "blargh" on it and some ancient crap I wrote years ago.</p>
<p>More recently, I needed to assemble a portfolio (both technical and musical) for types of work that I wanted to be doing, and this seemed like a good time to explore new options for site management and deployment. I'd seen <a href="https://getpelican.com">Pelican</a> pop up more and more on various technical sites, and have always preferred the speed and simplicity of maintaining a static site over a PHP-driven blog/CMS. But I hated hand-coding HTML, and markup editors tended to be engineered for LaTeX-oriented processing.</p>
<p>I enjoy Markdown because it doesn't get in the way, so I can crank out docs pretty quickly using it. Since Pelican supports Markdown as an input format, it seemed like a good place to start.</p>
<p>The net result is that andrewboring.com is now a <a href="https://getpelican.com">Pelican</a> site, and is hosted at Github pages.</p>
<p>Benefits to me:</p>
<ul>
<li>I can extend it with Python. Easy peasy lemon squeezy!</li>
<li>I can use Javascript for any interactivity, or to interface with API-based backends.</li>
<li>Unlike Wordpress, I just <em>write</em>. I don't have to use some editor named "Gutenberg."</li>
</ul>
<p><img alt="The Gutes!" src="https://andrewboring.com/media/steve-guttenberg.jpg">
<em>This doesn't look like a Wordpress editor.</em></p>
<p>Benefits to you:</p>
<ul>
<li>Static content can be more secure. Fewer attack vectors for a bad actor to upload malicious content for distribution.</li>
<li>Static content can be served faster than database-driven, dynamic content. There is no need for web-applications to drive stateless content.</li>
<li>I can focus on NOT having to wrangle Wordpress into doing crap it was never designed to do.</li>
</ul>Background Work2019-09-15T00:00:00-04:002019-09-15T00:00:00-04:00Andrew Boringtag:andrewboring.com,2019-09-15:/content/9413.html<h3>Lights! Camera! Action!</h3>
<p>The Big Screen came to Georgia years ago, and the film industry has just been getting better. So good, that you too can become a <strong>STAR</strong>, just like 9413, a Hollywood Extra:</p>
<p><span class="videobox">
<iframe width="640" height="390"
src='https://www.youtube.com/embed/xnAEHFhxuQI'
frameborder='0' webkitAllowFullScreen mozallowfullscreen
allowFullScreen>
</iframe>
</span></p>
<p>In the meantime, I registered with <a href="https://www.centralcasting.com">Central Casting</a>, who provides Background Talent for movie and …</p><h3>Lights! Camera! Action!</h3>
<p>The Big Screen came to Georgia years ago, and the film industry has just been getting better. So good, that you too can become a <strong>STAR</strong>, just like 9413, a Hollywood Extra:</p>
<p><span class="videobox">
<iframe width="640" height="390"
src='https://www.youtube.com/embed/xnAEHFhxuQI'
frameborder='0' webkitAllowFullScreen mozallowfullscreen
allowFullScreen>
</iframe>
</span></p>
<p>In the meantime, I registered with <a href="https://www.centralcasting.com">Central Casting</a>, who provides Background Talent for movie and tv productions. Yep. I stand in the background, pantomiming conversations, and try not to be too noticeable. I'm LIVING THE DREAM, just like 9413.</p>
<p>Some of the productions I've worked on (uncredited): </p>
<ul>
<li>Dr. Bird's Advice for Sad Poets (2018)</li>
<li>The Gifted (2018)</li>
<li>Beauty and The Baker (2019)</li>
<li>McGuyver</li>
<li>Dynasty (2019)</li>
<li>Genius: Aretha Franklin (2020)</li>
</ul>
<p>Sorry. No spoilers allowed.</p>Music2019-09-15T00:00:00-04:002019-09-15T00:00:00-04:00Andrew Boringtag:andrewboring.com,2019-09-15:/content/music.html<p>“Music gives a soul to the universe, wings to the mind, flight to the imagination and life to everything.” ― Plato</p>
<h2>Solo Projects</h2>
<p>From <a href="https://www.rpmchallenge.com/index.php?option=com_content&view=article&id=2&Itemid=193">RPM Challenge</a> 2017:
<iframe width="100%" height="350" scrolling="yes" frameborder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/305657399&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_te50er=true"></iframe></p>
<p>Listen to more music on my <a href="https://soundcloud.com/andrewboring/sets">SoundCloud page</a>.</p>
<h2>Music from DP3</h2>
<iframe width="100%" height="350" scrolling="yes" frameborder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/333221419&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true"></iframe>
<p>https://soundcloud.com/dp3online/sets/live-at-the-five-spot</p>
<iframe width="100%" height="350" scrolling="yes" frameborder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/333220305&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true"></iframe>Videos and Presentations2019-09-15T00:00:00-04:002019-09-15T00:00:00-04:00Andrew Boringtag:andrewboring.com,2019-09-15:/content/videos.html<p>A selection of video featuring yours truly.</p>
<h2>SwiftStack Cassandra Integration Demo</h2>
<p>This was a technical demo I created for the SwiftStack Systems Engineering team for using SwiftStack as a Cassandra keyspace backups.<br>
<video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/SwiftStack+Cassandra+Backup+Demo.mp4">
</video></p>
<h2>Internap Marketing Videos</h2>
<p><em>Note: The original YouTube videos have been marked "private" by the company's new owners. The …</em></p><p>A selection of video featuring yours truly.</p>
<h2>SwiftStack Cassandra Integration Demo</h2>
<p>This was a technical demo I created for the SwiftStack Systems Engineering team for using SwiftStack as a Cassandra keyspace backups.<br>
<video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/SwiftStack+Cassandra+Backup+Demo.mp4">
</video></p>
<h2>Internap Marketing Videos</h2>
<p><em>Note: The original YouTube videos have been marked "private" by the company's new owners. The videos below are copies.</em></p>
<h3>Ask the Expert: OpenStack and its Advantages</h3>
<p><video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/Ask+the+Expert+OpenStack+and+its+Advantages.mp4">
</video></p>
<h3>Bare-Metal AgileSERVER Powered by OpenStack</h3>
<p><video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/Bare-Metal+AgileSERVER+Powered+by+OpenStack.mp4">
</video></p>
<h3>Open Source Cloud and its Advantages</h3>
<p><video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/Open+Source+Cloud+and+its+Advantages.mp4">
</video></p>
<h3>OpenStack Bare-Metal AgileSERVERs Beta</h3>
<p><video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/OpenStack+Bare-Metal+AgileSERVERs+Beta.mp4">
</video></p>
<h3>Backups and Backup Strategies</h3>
<p><video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/Backups+and+Backup+Strategies.mp4">
</video></p>
<h3>Horizontal vs Vertical Scaling</h3>
<p><video id="" class="video-js vjs-default-skin" controls
preload="auto" width="683" height="384" poster=""
data-setup="{}">
<source src="https://d19pl2q67l2eit.cloudfront.net/Vertical+vs.+Horizontal+Scaling.mp4">
</video></p>Voiceover Work2019-09-15T00:00:00-04:002019-09-15T00:00:00-04:00Andrew Boringtag:andrewboring.com,2019-09-15:/content/voice.html<p>I do voice work on occasion.</p>
<h2>Internap Marketing Video</h2>
<h3>How To: Provisioning Bare Metal Servers on OpenStack</h3>
<p><span class="videobox">
<iframe width="426" height="240"
src='https://www.youtube.com/embed/1htJ-EUZjb0'
frameborder='0' webkitAllowFullScreen mozallowfullscreen
allowFullScreen>
</iframe>
</span></p>Quora: Organizing Metadata in a Data Lake2019-05-03T00:00:00-04:002019-05-03T00:00:00-04:00Andrew Boringtag:andrewboring.com,2019-05-03:/content/quora-organizing-metadata-in-a-data-lake.html<p><span class='quora-content-embed' data-name='How-do-you-store-the-metadata-for-all-the-data-in-your-centralized-data-lake/answer/Andrew-Boring'>Read <a class='quora-content-link' data-width='560' data-height='260' href='https://www.quora.com/How-do-you-store-the-metadata-for-all-the-data-in-your-centralized-data-lake/answer/Andrew-Boring' data-type='answer' data-id='138346122' data-key='4f4e694b45c119fbbd18e7bb1b2ea308' load-full-answer='True' data-embed='pWnkSGD'><a href='https://www.quora.com/Andrew-Boring'>Andrew Boring</a>'s <a href='/How-do-you-store-the-metadata-for-all-the-data-in-your-centralized-data-lake#ans138346122'>answer</a> to <a href='/How-do-you-store-the-metadata-for-all-the-data-in-your-centralized-data-lake' ref='canonical'><span class="rendered_qtext">How do you store the metadata for all the data in your centralized data lake?</span></a></a> on <a href='https://www.quora.com'>Quora</a><script type="text/javascript" src="https://www.quora.com/widgets/content"></script></span></p>Quora: Are there any cloud storage that support ‘deniable encryption’?2019-02-26T00:00:00-05:002019-02-26T00:00:00-05:00Andrew Boringtag:andrewboring.com,2019-02-26:/content/quora-are-there-any-cloud-storage-that-support-deniable-encryption.html<p><span class='quora-content-embed' data-name='Are-there-any-cloud-storage-that-support-deniable-encryption/answer/Andrew-Boring'>Read <a class='quora-content-link' data-width='560' data-height='260' href='https://www.quora.com/Are-there-any-cloud-storage-that-support-deniable-encryption/answer/Andrew-Boring' data-type='answer' data-id='124886110' data-key='a445ef4723a1fdb697d7d1969980ab09' load-full-answer='True' data-embed='pWnkSGD'><a href='https://www.quora.com/Andrew-Boring'>Andrew Boring</a>'s <a href='/Are-there-any-cloud-storage-that-support-deniable-encryption#ans124886110'>answer</a> to <a href='/Are-there-any-cloud-storage-that-support-deniable-encryption' ref='canonical'><span class="rendered_qtext">Are there any cloud storage that support ‘deniable encryption’?</span></a></a> on <a href='https://www.quora.com'>Quora</a><script type="text/javascript" src="https://www.quora.com/widgets/content"></script></span></p>