improve repo clone performance & root experience

FOCUS:

remove/modify operations which incur a big performance hit, like cloning all file contents versus downloading as used.

support for git command options clone and config to coincide with “terminal-first” effort mentioned elsewhere.

RESULTS:

Roughly, load time is down to 33.33% of prior and initial storage usage is 2% of prior. Caveats exist and I won’t expand on those right now.

image

TASKS:

  • remove files from service manifest that don’t need to be there
  • move templates to their own repo
  • keep repo files on github until needed
  • service worker build system (slightly not planned)
  • init modules refactor (each major component to its own folder)
  • DO NOT: download all github files when cloning repo
  • DO NOT: save files to wrong location with empty contents (BUG SOMEWHERE CAUSES THIS)
  • git clone: download a repo that doesn’t already exist
  • when git commit fails should not blow away changes
  • git list
  • git open/close
  • “root” terminal; change the way terminal determines currently loaded service
  • git diff should not show .git folder changes
  • git config, esp at root (for intial clone and storing user info/creds)
    • local file update
    • global file update
    • show file explorer at root
    • update service tree (so that new file appears in tree)
  • service worker should:
    • ignore contents of .git folder
    • use .git/config for credentials, local first then global (did this with terminal, not sw)
  • file “##PLACEHOLDER##”
    • show DELETED for deleted files (not a diff)

WORKED ON, BUT UNPLANNED:

  • editor
  • tree icons:
    • yaml
    • editorconfig
    • license
    • .gitignore
    • .git/config
    • .profile (shell)
  • favicon for beta
  • isometric git exploration (see below)
  • build vs transpile (see below)
  • plugins/extensions exploration
  • tree changed files indicator icons are still jumping on Mac on hover/exit
  • cleaned up + notes about C# compile/template
  • cleaned up blog files + iterated on blog style
  • added versions and info to plugins/vendor libs
  • hid the following:
    • tree menu: no real need for this
    • service map icon: this feature is not done and could be confusing in its current state
    • settings icon: terminal commands for all this now, plus it’s confusing
    • terminal menu: no need for this currently

isometric git exploration

I had a fun time talking myself into and out of integrating isometric git.

I’ll say first what motivated me to explore isogit. This system doesn’t leverage any library for its git or file system abstractions. Nuff said? I feel like this point could and already has been labored. Got a meme about wheel invention we can throw at this paragraph?

That being said, I’m not convinced that what git does is also what an web/cloud IDE should do. I’m looking at you, .git folder. It all feels way more verbose than I think appropriate here, at least for now.

Here’s what I’d like to do, indeed why I am exploring this:

  1. grab all the metadata about files and dirs in repo
  2. only download the files when they are used
  3. offer an option to go offline (download all the files)

And here’s what I understand about what git clone does:

  1. grab all the metadata about files, dirs, and history of changes in repo
  2. download all the files in their current state and (somehow) all info needed to reconstruct history

git and isogit grab “everything” and I only want to grab what is needed unless I have to for offline mode. But that’s not all they do. They also offer a mature interface and a sort of workflow for doing all that. With fiug, I do not have that mature interface implemented. And I’d really like to NOT have to re-invent that.

isogit “comes with” a file system as well, lightning-fs. This filesystem abstraction provides a good share of what one might need when using a filesystem. These should look familiar to anyone that has fooled around in a unix terminal:

  • readFile
  • writeFile
  • unlink
  • readdir
  • mkdir
  • rmdir
  • stat
  • lstat
  • readlink
  • symlink
  • chmod

fiug has a good portion of these, but it’s missing symlinks, file modes, and file status. Repeat what was said about not reinventing.

Here’s why I did not take the plunge and commit fiug to either isogit, lightning-fs, or even browserFS. Changing what I have to support what I mentioned I’d like to do above means a small change. And commiting to any of the alternatives means a bigger change. So it makes more sense to do the small change and revisit the large change after I take off the execution hat and put back on the product hat.

I don’t feel terribly great about this, but I’m confident that it’s the right way to move forward. Make sure that progress on the thing at hand is made, then decide if the future awesome is worth it later. This may mean that I move git pull out of this version, but I will definitely have git clone and it will work as expected unless you look under the hood.

build versus transpile

I’ve been working on supporting two similar features: import resolution in workers and module bundling. I’ll detail both of these and then talk about what I think is interesting about both taken together.

First, a little background on imports. I’ve used es6 modules as much as possible in writing browser code for fiug. I’ve been surprised how possible it is to write and run code in the browser without the need to transpile or build. Imagine, writing front-end code without needing a backend to translate for you! My thinking is that transpiling can happen as part of the last step before final release, if not never. We can argue whether this is a pipe dream or not, but I can say for sure it breaks down in the web worker context. In short, Web workers struggle with importing in a way that feels comfortable with me. In searching for a better feel, I reach the need to rewrite code that imports other modules. Service worker and babel to the rescue. With this solution I can write import statements that look like what I’d write with the backend build system. I can use import maps almost like a package.json. To put it briefly, the dream is alive and it feels pretty solid.

The service worker story is a bit different, though. Since the SW is what does the transpiling for workers, it’s a case of who transpiles the transpiler. I decided I would create a build script for the service worker and check the output of that in ahead of time. I arrived at using rollup. To be frank, the rollup experience was much more pleasant. With rollup, I solve almost the same problem by writing a plugin that resolves modules. With babel, I have to dig deeper into the weeds by writing a plugin that is closer to code parsing. This begs a question.

Why not just use rollup for both?

The best answer I can currently give is that babel is closer to the “metal” than rollup. To me, that means it’s the better solution for on-the-fly translation of code. Still, I prize the notion of reducing cognitive overhead and maintenance cost. I struggle to balance cogntivie load versus performance. That said, I’ll live with doing both until I have a better answer.

I thought it was an interesting thing to think about regardless. I’ve tried to keep it short, but I am happy to offer more detail if asked. I’m still working on improving my technical communication skills.

Here’s the solution I am settling on:

  1. service worker has it’s own build using rollup
    • can build within fiug
    • should also work through a github action
  2. workers are ran using “node” command
    • “node” will access /-/worker/{path to script} to get a transpiled worker
    • service worker will use babel to resolve all depenendencies (will rewrite specifiers)

As I dig deeper into this, I see a need to somehow cache dependencies into a vendor file that is somehow checked in. I saw this issue with skypack.dev not finding files which terminal needs. This resulted in terminal showing blank. This may be solved as I work out builds for each of the major components: tree, editor, and terminal.

Also, I see an issue with editor cache not being invalidated when service-worker build script is ran. This makes it look like the build script did not run. https://beta.fiug.dev/service/change needs an option to invalidate editor cache, I think.