Skip to main content

Speeding up bash startup #

From 600ms to 14ms

Previously: How long do commands run?

VSCode has had a fancy terminal IntelliSense for some time now. For some reason, it only worked on my macOS laptop, but not on my Linux machine. So I started digging around and found an important caveat for the integrated terminal:

Note that the script injection may not work if you have custom arguments defined in the terminal profile, have enabled Editor: Accessibility Support, have a complex bash PROMPT_COMMAND, or other unsupported setup.

Turns out that my use of bash-preexec messed up the PROMPT_COMMAND enough that VSCode couldn't inject itself properly.

Now as I described in the previous post, I'm only really using bash-preexec to measure the run time of commands. So I used ChatGPT 5.2 and Claude Opus 4.5 to help me work through my .bashrc to remove that constraint.

First, we keep track of whether we're in the prompt (we don't want to time those commands) and we separately "arm" the timer after the prompt is drawn (so we can time things after the next command runs).

# at the top
__cmd_start_us=0
__cmd_timing_armed=0
__in_prompt=0

__timer_arm() { __cmd_timing_armed=1; }
__timer_debug_trap() {
  [[ $__in_prompt -eq 1 ]] && return 0
  [[ $__cmd_timing_armed -eq 1 ]] || return 0
  __cmd_timing_armed=0
  local s=${EPOCHREALTIME%.*} u=${EPOCHREALTIME#*.}
  __cmd_start_us="${s}${u:0:6}"
}

trap '__timer_debug_trap' DEBUG
__s=${EPOCHREALTIME%.*}
__u=${EPOCHREALTIME#*.}
__cmd_start_us="${__s}${__u:0:6}"
unset __s __u

# ...

PROMPT_COMMAND="__prompt_command; __timer_arm"

The trap bit is clever and does most of the heavy lifting.

Once I got this working with my PS1 (see below), I asked Claude for any other improvements it could think of. I did this 3 times and incorporated all of its suggestions.

The main things I changed were to lazy-load completions and other imports. This brought the shell startup time down from 600ms to 14ms which I definitely notice.

__load_completions() {
  unset -f __load_completions
  if ! shopt -oq posix; then
    if [ -f /usr/share/bash-completion/bash_completion ]; then
      . /usr/share/bash-completion/bash_completion
    elif [ -f /etc/bash_completion ]; then
      . /etc/bash_completion
    fi
  fi

  # nvm
  [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

  # uv
  eval "$(command uv generate-shell-completion bash)"
}
complete -D -F __load_completions # trigger on first tab-complete

# ...

# https://github.com/git/git/blob/master/contrib/completion/git-prompt.sh
__git_ps1() { unset -f __git_ps1; . ~/.git-prompt.sh; __git_ps1 "$@"; }

# ...

export NVM_DIR="$HOME/.nvm"
nvm() { unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; nvm "$@"; }
node(){ unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; node "$@"; }
npm() { unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; npm "$@"; }
npx() { unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; npx "$@"; }

Then there were some quality-of-life improvements:

HISTCONTROL=ignoreboth:erasedups
shopt -s histappend histverify # append and expand history file
HISTTIMEFORMAT="%F %T " # timestamp entries
HISTSIZE=10000
HISTFILESIZE=20000

# ...

shopt -s globstar  # let '**' match 0 or more files and dirs
shopt -s cdspell   # autocorrect minor typos in cd
shopt -s autocd    # type directory name to cd into it

The biggest of these was using fzf:

__fzf_lazy_init() { unset -f __fzf_lazy_init; eval "$(fzf --bash)"; }
bind '"\C-r": "\C-x1\C-r"'
bind '"\C-t": "\C-x1\C-t"'
bind '"\ec": "\C-x1\ec"'
bind -x '"\C-x1": __fzf_lazy_init'

This is another lazy-loaded bit, but what this gives you is a much better history search (CTRL+R), file search (CTRL+T), and better cd (ALT+C).

Here's what it looks like all together:

__cmd_start_us=0
__cmd_timing_armed=0
__in_prompt=0

__timer_arm() { __cmd_timing_armed=1; }
__timer_debug_trap() {
  [[ $__in_prompt -eq 1 ]] && return 0
  [[ $__cmd_timing_armed -eq 1 ]] || return 0
  __cmd_timing_armed=0
  local s=${EPOCHREALTIME%.*} u=${EPOCHREALTIME#*.}
  __cmd_start_us="${s}${u:0:6}"
}

trap '__timer_debug_trap' DEBUG
__s=${EPOCHREALTIME%.*}
__u=${EPOCHREALTIME#*.}
__cmd_start_us="${__s}${__u:0:6}"
unset __s __u

###

case $- in
    *i*) ;;
      *) return;;
esac

HISTCONTROL=ignoreboth:erasedups
HISTTIMEFORMAT="%F %T "  # timestamp entries
HISTSIZE=10000
HISTFILESIZE=20000
shopt -s histappend histverify # append and expand history file
shopt -s checkwinsize globstar

[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
  debian_chroot=$(cat /etc/debian_chroot)
fi

case "$TERM" in
  xterm-color|*-256color) color_prompt=yes;;
esac

if [ -n "$force_color_prompt" ]; then
  if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
	  color_prompt=yes
  else
    color_prompt=
  fi
fi

if [ "$color_prompt" = yes ]; then
  PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
  PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
  PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
  ;;
*)
  ;;
esac

if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'
alias gitroot='cd $(git rev-parse --show-toplevel)'
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# User defined

shopt -s cdspell   # autocorrect minor typos in cd
shopt -s autocd    # type directory name to cd into it

__load_completions() {
  unset -f __load_completions
  if ! shopt -oq posix; then
    if [ -f /usr/share/bash-completion/bash_completion ]; then
      . /usr/share/bash-completion/bash_completion
    elif [ -f /etc/bash_completion ]; then
      . /etc/bash_completion
    fi
  fi

  # nvm
  [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

  # uv
  eval "$(command uv generate-shell-completion bash)"
}
complete -D -F __load_completions # trigger on first tab-complete


# Terminal Prompt

# From: https://github.com/metaist/brush
BR_RESET="\e[0;0m"
BR_GREEN="\e[32m"
BR_YELLOW="\e[33m"
BR_BGRED="\e[41m"

# https://github.com/git/git/blob/master/contrib/completion/git-prompt.sh
__git_ps1() { unset -f __git_ps1; . ~/.git-prompt.sh; __git_ps1 "$@"; }

VIRTUAL_ENV_DISABLE_PROMPT=true # we'll handle it ourselves
PROMPT_COMMAND="__prompt_command; __timer_arm"
__prompt_command() {
  local code="$?" # must be first
  local s=${EPOCHREALTIME%.*} u=${EPOCHREALTIME#*.}
  local now_us="${s}${u:0:6}"
  __in_prompt=1

  PS1="\[\e]0;\w\a\]\n" # set terminal title

  local venv=${VIRTUAL_ENV##*/}
  PS1+="${venv:+($venv) }" # venv name

  # add run time of previous command [error code in red]
  local dur_ms=$(( (10#$now_us - 10#$__cmd_start_us) / 1000 ))
  PS1+="${dur_ms}ms"
  if [[ "$code" != "0" ]]; then
    PS1+=" $BR_BGRED[err $code]$BR_RESET"
  fi

  PS1+="\n$BR_GREEN\u@\h$BR_RESET "   # user@host
  PS1+="$BR_YELLOW\w$BR_RESET"        # pwd
  PS1+="$(__git_ps1)"                 # git info
  PS1+="\n\$ "                        # cursor

  __in_prompt=0
}

__prepend_path() { [[ ":$PATH:" != *":$1:"* ]] && PATH="$1:$PATH"; }

# fzf
__fzf_lazy_init() { unset -f __fzf_lazy_init; eval "$(fzf --bash)"; }
bind '"\C-r": "\C-x1\C-r"'
bind '"\C-t": "\C-x1\C-t"'
bind '"\ec": "\C-x1\ec"'
bind -x '"\C-x1": __fzf_lazy_init'

# node/bun
export BUN_INSTALL="$HOME/.bun"
__prepend_path "$BUN_INSTALL/bin"

# node/pnpm
export PNPM_HOME="$HOME/.local/share/pnpm"
__prepend_path "$PNPM_HOME"

# node/nvm
export NVM_DIR="$HOME/.nvm"
nvm() { unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; nvm "$@"; }
node(){ unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; node "$@"; }
npm() { unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; npm "$@"; }
npx() { unset -f nvm node npm npx; . "$NVM_DIR/nvm.sh"; npx "$@"; }

# python
export PYTHONDONTWRITEBYTECODE=1

__prepend_path "$HOME/.local/bin"


πŸ“ Logging Sucks - Your Logs Are Lying To You (Boris Tane). Argues for passing a context object around and logging that object (with all the details you could possibly need) when something goes wrong. Extends the concept of structured logging to "wide events".




πŸ“ Why Stripe’s API Never Breaks: The Genius of Date-Based Versioning (Harsh Shukla). I got through most of this post before it was revealed that Stripe has a version-level description of which features were added to the API and adapters that convert inputs and outputs into the appropriate version level based on date. Very cool, but how do you handle security issues in versions? You options (as far as I can tell are):

  1. Announce you can no longer use a particular version. (Breaks "we support every version".)
  2. Change the behavior of the specific version and re-release with the same version number. (Breaks "this version has this particular behavior".)
  3. Some kind of automatic translation that says "this published version maps to this internal version".

In any case, it's all very nice, but unlikely to impact how most people will design versioned artifacts in the future.






πŸ“ On deathbed advice/regret (hazn; via Tyler Cowen). I agree with the main point of the post which is why I've usually taken deathbed regret and converted it into specific advice. For many years, I've had the following (lightly edited) list towards the top of my todo list:

  • Don't ignore your dreams
  • Don't work too much
  • Say what you think
  • Cultivate friendships
  • Be happy





cosmofy 0.2.1 #

Gotta patch 'em all!

cosmofy 0.2.1 is available. Lots of little bugs after this morning's big release.

If you have uv installed you can use cosmofy:

uvx cosmofy

# or install it
uv tool install cosmofy

Read more.

Release Notes 0.2.1 - 2025-12-24T18:35:27Z #

Fixed

  • #42 minor typos; added links in README
  • #58 only do the Windows dance if literally trying to replace sys.executable
  • #96 files not saving to cache
  • #97, #98 executable permissions for Cosmopolitan Python
  • #99 nothing being logged in binary build
  • #95 don't search for pyproject.toml when you only have --script to bundle
  • #104 error when using cosmofy fs ls with --sort none
  • #105 restore working directory after error when using cosmofy fs add with --chdir
  • #103 use context manager for urlopen even for HEAD

Changed

  • #106 cosmofy fs cat outputs raw bytes to sys.stdout instead of decoding to UTF-8

Added

  • #94 pure Python limitation to README
  • #107 clearer docstring for expand_glob; literal paths are returned exactly
  • #100 document that pythonoid.run_python intentionally destroys sys.argv

Security

  • #102 call validate_url at the start of all the download_* functions
  • #109 reject paths with null bytes
  • #108 don't follow circular symlinks endlessly

cosmofy 0.2.0 #

Now using uv!

cosmofy 0.2.0 is available. So many things came together for this release:

  1. Three open source developers I follow (William McGuan, Simon Willison, and Charlie Marsh) were all in a twitter thread where the concept of something like cosmofy was mentioned.
  2. I learned how to make CLIs sort of work the way Astral's tools do.
  3. I learned just how many crazy options GNU ls supports.

If you have uv installed you can use cosmofy:

uvx cosmofy

# or install it
uv tool install cosmofy

Read more.

Release Notes 0.2.0 - 2025-12-24T03:14:02Z #

This release represents a very large shift from bundling individual python files to using uv to bundle entire venv directories. The behavior of the CLI is now much more similar to uv in form and function.

Fixed

  • #14 typo in --help
  • #43 exclude uv artifacts from the bundle
  • #46 expand_globs to follow all (most?) of the shell rules
  • #48 getting only the project's console_scripts
  • #50 cosmofy bundle --script assumes venv in output
  • #52 avoiding clobbering logging on import
  • #56 used context handlers to close temporary directories
  • #58 self update on Windows
  • #59 assumption that Last-Modified will be present in headers
  • #61 replaced PathDistribution._path with appropriate fallbacks
  • #62 replaced assert for validation with actual raise on error
  • #66 needless pass in parsing args
  • #67 command names in logging
  • #70 incorrect shebang in bundle.py
  • #71 used context handlers to close ZipFile objects
  • #72 download progress indicator shouldn't divide by zero
  • #73 missing newline in bundle.py error message
  • #75 explicit return of default when we can't get the version from a file
  • #77 cosmofy fs cat usage string
  • #79 ZipFile2.now should not be at module-level
  • #92 use public API for SourceFileLoader

Changed

  • #16 update examples, readme, usage (HT @Pugio)
  • #28 replaced --debug with --quiet + --verbose
  • #29 DEFAULT_PYTHON_URL is now Cosmopolitan Python
  • #31 replaced --cache with --no-cache + --cache-dir
  • #33 refactored bundler.py into subcommands
  • #55 switched to JSON parsing uv version output
  • #57 switched to using os.replace for atomic file moves
  • #64 replaced removeprefix with better cross-platform approach
  • #79 moved progress indicator to stderr
  • #87 marked cosmofy updater commands as experimental

Added

  • #18 progress when downloading (HT @Pugio)
  • #32 cosmofy bundle --script
  • #33 cosmofy bundle
  • #33 cosmofy fs add
  • #33 cosmofy fs args
  • #33 cosmofy updater remove
  • #33 dry run banner
  • #35 cosmofy fs ls
  • #36 cosmofy fs cat
  • #37 cosmofy fs rm
  • #38 cosmofy fs add -f to overwrite files that exist
  • #39 --color global option
  • #40 cosmofy self update
  • #40 cosmofy self version
  • #49 --exact to uv sync for better exact specs
  • #60 timeouts to network calls
  • #81 note that concurrent removals are not supported
  • #83 missing docstrings
  • #86 log message that only GitHub URLs are inferred
  • #89 warning when adding absolute paths
  • #91 note about handling weird file names

Removed

  • #15 cosmo.yaml workflow; added gh command
  • #17 default .com extension
  • #24 support for python <= 3.9
  • #27 --clone is no longer supported
  • #30 --download is no longer supported

Security

  • #53 added validate_url to avoid non-HTTPS URLs
  • #53 added security considerations note
  • #54 removed shell=True in places that don't need it
  • #63 added sanitize_zip_path to avoid bad entry names

cosmofy fs ls: emulating GNU ls #

How much of GNU ls should I emulate in Python?

Previously: pythonoid, baton

When I was designing the low-level zip file manipulation tools for cosmofy, I wanted an easy way to see the contents of the bundle. We're so used to using ls for looking into directories that I thought it would be cool to emulate as much of ls as I could.

But then it turned out that ls has a crazy number of options. I actually went through them all and tried to figure out if it was possible to support them.

But then I realized this was insane. First, many of the options are just aliases for slightly more explicit options. Charlie Marsh would never have a -t that was an alias for --sort=time. Why should I?

In the end I decided to go with the most common options (sorting, list view), a couple that were easy to implement, and a few longer-form ones that cover most of the aliases.

I'm pretty happy with the way it turned out.


baton: emulating Astral's CLI approach #

How much of the Astral CLI theme can I emulate in Python?

Previously: pythonoid

Imitation is the highest form of flattery which is why as part of the cosmofy 0.2.0 release, I decided to change everything about how the CLI behaved to make it work more like the way the tools from Astral work.

I have a long-term plan for Astral to take over making Cosmopolitan Python apps. It's a long shot, but if they do, it'll be a huge win for cross-platform compatible executables. I also saw this popular issue that there should be a uv bundle command that bundles everything up.

To make it easier to adopt, I decided to make the interface follow Astral's style in three important ways:

  1. Subcommand structure: It's gotta be cosmofy bundle and cosmofy self update
  2. Colored output: Gotta auto-detect that stuff. Luckily, I had fun with brush years ago, so I know about terminal color codes.
  3. Global flags: Some of those flags gotta be global.
  4. Smart ENV defaults: smart defaults + pulling from environment variables to override.

Now I didn't start out wanting to build my own argument parser (really, I promise I didn't!). I tried going the argparse route (I even tried my own attrbox / docopt solution), but I had a few constraints:

  1. I really don't want 3rd party dependencies (even my own). cosmofy needs to stay tight and small.
  2. I want argument parsing to go until it hits the subcommand and then delegate the rest of the args to the subcommand parser.
  3. I want to pass global options from parent to child sub-parser as needed.

Together these pushed for a dedicated parser. This lets me write things like:

usage = f"""\
Print contents of a file within a Cosmopolitan bundle.

Usage: cosmofy fs cat <BUNDLE> <FILE>... [OPTIONS]

Arguments:
{common_args}
  <FILE>...                 one or more file patterns to show

  tip: Use `--` to separate options from filenames that start with `-`
  Example: cosmofy fs cat bundle.zip -- -weird-filename.txt

Options:
  -p, --prompt              prompt for a decryption password

{global_options}
"""


@dataclass
class Args(CommonArgs):
    __doc__ = usage
    file: list[str] = arg(list, positional=True, required=True)
    prompt: bool = arg(False, short="-p")

...

def run(args: Args) -> int:
  ...

cmd = Command("cosmofy.fs.cat", Args, run)
if __name__ == "__main__":
    sys.exit(cmd.main())

For the colored output, I took inspiration from William McGuan's rich which uses tag-like indicators to style text.

Mine is much worse and minimal, but it gets the job done for the bits of color I need.



πŸ“ How I Found Myself Running a Microschool (Kelsey Piper / Center for Educational Progress). Over the past 10 years I have migrated to essentially this view: you need direct instruction to get the basics and a foundation; you need to see people enact the values you want to transmit; and you need a strong motivating project to get you over the humps when the going gets tough.



πŸ“ Ideas Aren’t Getting Harder to Find (Karthik Tadepalli / Asterisk). Knowing what is causing productivity growth to start to slow is critical to selecting appropriate policies for how to get it going again. Karthik makes a good case for why the idea that "ideas are getting harder to find" is wrong and why it's more of a failure of the market to weed out bad ideas and promote good ones.




πŸ“ Justified (Bite code!). just is a Rust-based task runner and here's a good write up about its features. If I wasn't writing ds (my own task runner), I'd probably use just. I'll probably experimentally support some subset of just's syntax in ds.


πŸ“ Book Time #26: How I Read (Aaron Gordon / Book Time). My process for selecting books is very different from Aaron's, but it's nice to see someone else's process.

Mine:

  • I get recommendations from friends, colleagues, podcasts, and other books.
  • I add them to a list (currently an Amazon Wishlist). I prefer digital books over physical books because I can make them into audiobook (at 4x) and they take up very little physical space.
  • When I get to the end of the books I have in my queue, I sort the wishlist by price (lowest to highest) and buy everything under $5. That keeps me busy for about 3 months.
  • I take all the books that I've purchased and sort them by the number of pages (lowest to highest). This is my new reading queue.

There are only a couple of exceptions for me to deviate from this order:

  1. My wife recommends I read something next, especially as it may impact our kids. This bumps the book to the top of the list (both for purchase and reading).
  2. I'm about to meet someone who recently published a book. There have been a few times where I got through their most-recently published book the day before meeting them. HT: Ramit Sethi who teaches the importance of being overly prepared for certain kinds of meetings.

I often get asked how much I retain from the audio versions of books. My informal tests show no more or less than most people when they read books. I don't remember every detail, but neither do most people. HT: Mike Masnick who introduced me to the idea of listening to podcasts at greater than 2x.




TIL: importlib.metadata.version #

Apparently __version__ can be dynamic now.

While I was updating my standard pyproject.toml to use dependency groups, I switched to using uv as my build backend (from setuptools). Like all the other design choices, it has very sensible defaults with good overrides when you need them.

One thing that it doesn't support is dynamic versions (i.e. reading __version__ from __init__.py). Charlie Marsh explains:

Using dynamic metadata for things that are actually just static lookups feels like the wrong tradeoff.

Like version = ["dynamic"] to read the version from __init__.py.

As soon as you have dynamic metadata, you need to install dependencies and run Python just to get that info.


Static metadata rules. Just write the version out twice!

This makes sense, but I really didn't want to write out the version twice. Luckily, uv has a nice uv version command to change/bump the version number in the pyproject.toml, so I started thinking about how I'd also change the version in the __init__.py at the same time.

I asked ChatGPT what the best approach was and it suggested something quite different: don't write the value in __init__.py at all.

from importlib.metadata import version, PackageNotFoundError
try:
    __version__ = version("package-name") # from pyproject.toml: project.name
except PackageNotFoundError:
    __version__ = "0.0.0" # aka unknown version

I have since noticed that this was also the suggestion Adam Johnson had in the same thread.

One minor concern I have is whether this works in a cosmofy build, but that will be a separate exploration.

View all posts