This is a debugging session for spack. The issue is that the self.stage.source_path changes (unexpectedly) in the middle of do_stage, and it's not clear why this happens. I'm at the HEAD of the repo, just pulled the fix to remove the empty line:
commit b92fe5f248892069674de2f090e4da879b5c83d4 (HEAD -> update/package-singularity, origin/update/package-singularity)
Author: Vanessa Sochat <vsochat@stanford.edu>
Date: Wed Apr 24 09:58:08 2019 -0400
removing blank link
Signed-off-by: Vanessa Sochat <vsochat@stanford.edu>
Before I jump in, here are some things to point out.
- For the MakefilePackage, whenever we call a use of self.build_directory it returns self.stage.source_path, but it's also a property, so (I think) this would mean both act as functions (and the self.path.stage_path could be changed silently. If you look at the stages (install, check, etc.) the fact that they all use this build directory could be an issue. Is the variable supposed to change?
- And it follows that whenever we call self.stage.source_path as I mentioned yesterday, it's going to change! I haven't used property decorators a ton, but this makes me uncomfortable because it's not clear when it's going to decide to change. It assumes that either I am aware of my filesystem not changing, or I'm aware of my filesystem changing (and I want the variable to change with it).
I think the issue comes down to using these @property dectorators, and thus having a variable that changes silently.
I want to make sure the stage is actually cleaned (these are the places that I know of)
$ ls -l var/spack/stage/
singularity-develop-7gbfecp3wbbrr7z3crhv3vm3tnbgjwvh
spack-stage-fxis13he
spack-stage-ej9jzy21
$ bin/spack clean -a
==> Removing all temporary build stages
==> Removing cached downloads
==> Removing cached information on repositories
==> Removing python cache files
$ ls var/spack/stage/
$ ls /tmp/vanessa/spack-stage/
First I'll also set a breakpoint at the first line of do_stage:
# Unpack the tarball as usual, then move the src dir into# its home within GOPATH.
def do_stage(self, mirror_only=False):
import pdb;pdb.set_trace()
super(Singularity, self).do_stage(mirror_only)
source_path = self.stage.source_path
if not os.path.exists(self.singularity_gopath_dir):And uninstall, install
$ bin/spack uninstall -a -y singularity
$ bin/spack install singularity@3.1.1
Nothing is defined yet:
==> No binary for singularity found: installing from source
> /tmp/spack/var/spack/repos/builtin/packages/singularity/package.py(56)do_stage()
-> source_path = self.stage.source_path
(Pdb) p self.stage.source_path
None
(Pdb) p self.path
None
(Pdb) p self
<spack.pkg.builtin.singularity.Singularity object at 0x7fe886ee4550>
I think we are here given the first message about installing from source.
The stage is also happening in /tmp (and we verified this was cleaned out before starting)
(Pdb) self.stage.path
'/tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla'
(Pdb)
Can also verify nothing is currently there! I think the client entrypoint calls do_install first so that we are at the end of this function. I'm not totally clear where the "spec" get's instantiated. But I can take a step and then we see the download start up.
(Pdb) n
==> Fetching https://github.com/sylabs/singularity/releases/download/v3.1.1/singularity-3.1.1.tar.gz
######################################################################### 100.0%
==> Staging archive: /tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/singularity-3.1.1.tar.gz
==> Created stage in /tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla
> /tmp/spack/var/spack/repos/builtin/packages/singularity/package.py(57)do_stage()
-> if not os.path.exists(self.singularity_gopath_dir):
And /tmp is now populated with the extracted package:
$ ls /tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla
singularity singularity-3.1.1.tar.gz
This is now correct, because it's using the first entry of os.listdir that is a folder.
(Pdb) self.stage.source_path
'/tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/singularity'
That makes sense, it's the first/only subdir of the staging dir. All the expected stuff
is in singularity
$ ls /tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/singularity
CHANGELOG.md dist Gopkg.toml makeit scripts
cmd docs INSTALL.md mconfig singularity.spec
CONTRIBUTING.md etc internal mlocal vendor
CONTRIBUTORS.md examples LICENSE-LBNL.md pkg VERSION
COPYRIGHT.md Gopkg.lock LICENSE.md README.md
(Pdb) n
> /home/ghartzell/spack/var/spack/repos/builtin/packages/singularity/package.py(61)do_stage()
-> mkdirp(self.sylabs_gopath_dir)
(Pdb) p self.stage.source_path
'/home/ghartzell/spack/var/spack/stage/singularity-3.1.1-kuf7bim7pigb6flgtwcbsimkxietlwpr/singularity'
(Pdb)
And here is the bug again - after the shutil call, it changes to src.
-> tty.debug("Moving {0} to {1}".format(
(Pdb) self.stage.source_path
'/tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/singularity'
(Pdb) n
> /tmp/spack/var/spack/repos/builtin/packages/singularity/package.py(59)do_stage()
-> self.stage.source_path, self.singularity_gopath_dir))
(Pdb) self.stage.source_path
'/tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/singularity'
(Pdb) n
> /tmp/spack/var/spack/repos/builtin/packages/singularity/package.py(60)do_stage()
-> mkdirp(self.sylabs_gopath_dir)
(Pdb) self.stage.source_path
'/tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/singularity'
(Pdb) n
> /tmp/spack/var/spack/repos/builtin/packages/singularity/package.py(61)do_stage()
-> shutil.move(source_path,
(Pdb) self.stage.source_path
'/tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/singularity'
(Pdb) n
> /tmp/spack/var/spack/repos/builtin/packages/singularity/package.py(62)do_stage()
-> self.singularity_gopath_dir)
(Pdb) self.stage.source_path
'/tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/src'
Any call to self.stage.source_path is going to run this function It seems logical to me - we can look in the /tmp folder and now see that (since we mkdir -p the path starting with src) it's now there:
$ ls /tmp/spack/var/spack/stage/singularity-3.1.1-3n7skikliv4pd6olbouxjtjsdl2j4vla/
singularity singularity-3.1.1.tar.gz src
Let's be silly and open up python in the directory and run the function. First I want to check the fetcher variable - it's actually the kind that would trigger this if statement:
if isinstance(self.fetcher, fs.URLFetchStrategy):
if not self.fetcher.expand_archive:
return self.pathHowever, it's in a list! So the entire thing is skipped over (is this intentional, perhaps it's done after the url is fetched so it doesn't trigger the statement and get fetched again?)
(Pdb) p self.fetcher[0]
URLFetchStrategy<https://github.com/sylabs/singularity/releases/download/v3.1.1/singularity-3.1.1.tar.gz>
(Pdb) p self.fetcher[0].expand_archive
True
And here is the bug. Knowing that this function returns the first directory, we cannot knowingly create a new one (src) and not expect to be called. The differences that we see between hosts machines are likely do to differences in the ordering of the listing of files.
In [1]: importosIn [2]: path=os.getcwd()
In [3]: forpin [os.path.join(path, f) forfinos.listdir(path)]:
...: ifos.path.isdir(p):
...: print(p)
...:
/tmp/vanessa/spack-stage/spack-stage-d56my3dl/src/tmp/vanessa/spack-stage/spack-stage-d56my3dl/singularityAnd this is why my fix to grab the original folder works. I don't want to comment on the design of the software, but for this pull request we can't rely on this changing variable.