Re: attempting hg clone under 1GB

[matthew sporleder <msporleder%gmail.com@localhost>]

On Tue, Aug 14, 2018 at 7:58 AM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
> On Sun, Aug 12, 2018 at 7:18 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>> On Sat, Aug 11, 2018 at 8:02 PM, matthew sporleder <msporleder%gmail.com@localhost> wrote:
>>> On Sat, Aug 11, 2018 at 5:36 PM, Joerg Sonnenberger <joerg%bec.de@localhost> wrote:
>>>> On Sat, Aug 11, 2018 at 12:06:59PM -0400, matthew sporleder wrote:
>>>>> Hi I attempted to run `hg clone https://bitbucket.org/netbsd/src` it
>>>>> ran for over an hour and then died, maxing out around 560MB.
>>>>
>>>> How did it die? Read error? It seems like Atlassian has some internal
>>>> problems with their media servers at times, typically showing as short
>>>> read errors. That should not affect cloning from anonhg.n.o.
>>>
>>> Like this:  I wonder if I ran out of disk space
>>>
>>> dev $ hg clone https://bitbucket.org/netbsd/src
>>> destination directory: src
>>> applying clone bundle from
>>> https://api.media.atlassian.com/file/8dee2882-6e78-4c5c-aaa2-75ca6de55ff7/binary?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIwYjE4ZTQzMC0yMDE3LTQ4NDMtOTZmMC02YTUyZWM3ZTE5ZGMiLCJhY2Nlc3MiOnsidXJuOmZpbGVzdG9yZTpmaWxlOjhkZWUyODgyLTZlNzgtNGM1Yy1hYWEyLTc1Y2E2ZGU1NWZmNyI6WyJyZWFkIl19LCJuYmYiOjE1MzM5OTkxMDcsImV4cCI6MTUzMzk5OTUyN30.ODA2D9DDbggs8pY81Oc49knJsA8mLLrN5MksksqTJ1s&client=0b18e430-2017-4843-96f0-6a52ec7e19dc
>>> adding changesets
>>> adding manifests
>>> adding file changes
>>> files [=====>
>>>                             ]  27985/402501 56m37s^R
>>> files [=====>
>>>                             ]  28026/402501 56m35s^R
>>> transaction abort!
>>>
>>>
>>> rollback completed
>>> abort: stream ended unexpectedly (got 0 bytes, expected 4)
>>
>>
>> Okay I tried again and, on the first try, my entire VM crashed!  (uh oh)
>>
>> The second one died (using anonhg this time)
>>
>> dev $ time hg clone https://anonhg.netbsd.org/src
>> destination directory: src
>> applying clone bundle from
>> https://cdn.NetBSD.org/_bundles/src/66ce2db9a56456a792bf238b5fe618142163cb79.zstd.hg
>> adding changesets
>> adding manifests
>> adding file changes
>> added 320428 changesets with 2162086 changes to 394784 files (+409 heads)
>> finished applying clone bundle
>> searching for changes
>> adding changesets
>> adding manifests
>> transaction abort!
>> rollback completed
>> abort: stream ended unexpectedly (got 0 bytes, expected 4)
>
> So even with more memory (swap) and free disk I can't get this to
> finish on my NetBSD box.
>
> The output is a little different each time:
> destination directory: src
> applying clone bundle from
> https://cdn.NetBSD.org/_bundles/src/3c77078a71f38b288632c2fe115c8a9e1a07b750.zstd.hg
> adding changesets
> adding manifests
> adding file changes
> added 363864 changesets with 2204519 changes to 407532 files (+422 heads)
> transaction abort!
> rollback completed
> abort: stream ended unexpectedly (got 0 bytes, expected 4)
>
> ---
>
> I *was* able to get the clone to complete on my big macbook pro.



The following made this work in ~/.hgrc:
[ui]
clonebundleprefers = COMPRESSION=bzip2

I stumbled on it because my osx client (older 4.0) was using bz instead of zstd

In terms of memory I ran over 600MB and then went through a weird
cycle of 50MB -> 500MB repeated while it did various patches

Matt