Bumped version

* Add a discord link in sidebar

* Added discord link to README

README.md

@@ -15,7 +15,8 @@
 <div align="center">
 
 [![](https://img.shields.io/github/license/kieraneglin/pinchflat?style=for-the-badge&color=ee512b)](LICENSE)
-[![](https://img.shields.io/github/v/release/kieraneglin/pinchflat?style=for-the-badge)](https://github.com/kieraneglin/pinchflat/releases)
+[![](https://img.shields.io/github/v/release/kieraneglin/pinchflat?style=for-the-badge&color=purple)](https://github.com/kieraneglin/pinchflat/releases)
+[![](https://img.shields.io/static/v1?style=for-the-badge&logo=discord&message=Chat&color=5865F2&label=Discord)](https://discord.gg/j7T6dCuwU4)
 [![](https://img.shields.io/github/actions/workflow/status/kieraneglin/pinchflat/lint_and_test.yml?style=for-the-badge)](#)
 [![](https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode&style=for-the-badge)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/kieraneglin/pinchflat)
 
@@ -33,6 +34,7 @@
   - [Portainer](#portainer)
   - [Docker](#docker)
   - [Environment Variables](#environment-variables)
+  - [A note on reverse proxies](#reverse-proxies)
 - [Username and Password (authentication)](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password)
 - [Frequently asked questions](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions)
 - [Documentation](https://github.com/kieraneglin/pinchflat/wiki)
@@ -143,7 +145,7 @@ If you change this setting and it works well for you, please leave a comment on
 ### Environment variables
 
 | Name                        | Required? | Default                   | Notes                                                                                                                                     |
-| --------------------------- | --------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| --------------------------- | --------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
 | `TZ`                        | No        | `UTC`                     | Must follow IANA TZ format                                                                                                                |
 | `LOG_LEVEL`                 | No        | `debug`                   | Can be set to `info` but `debug` is strongly recommended                                                                                  |
 | `BASIC_AUTH_USERNAME`       | No        |                           | See [authentication docs](https://github.com/kieraneglin/pinchflat/wiki/Username-and-Password)                                            |
@@ -152,8 +154,13 @@ If you change this setting and it works well for you, please leave a comment on
 | `ENABLE_IPV6`               | No        | `false`                   | Setting to _any_ non-blank value will enable IPv6                                                                                         |
 | `JOURNAL_MODE`              | No        | `wal`                     | Set to `delete` if your config directory is stored on a network share (not recommended)                                                   |
 | `TZ_DATA_DIR`               | No        | `/etc/elixir_tzdata_data` | The container path where the timezone database is stored                                                                                  |
-| `BASE_ROUTE_PATH`           | No        | `/`                       | The base path for route generation. Useful when running behind certain reverse proxies, but prefix must be stripped.            |
+| `BASE_ROUTE_PATH`           | No        | `/`                       | The base path for route generation. Useful when running behind certain reverse proxies - prefixes must be stripped.                       |
 | `YT_DLP_WORKER_CONCURRENCY` | No        | `2`                       | The number of concurrent workers that use `yt-dlp` _per queue_. Set to 1 if you're getting IP limited, otherwise don't touch it           |
+| `ENABLE_PROMETHEUS`         | No        | `false`                   | Setting to _any_ non-blank value will enable Prometheus. See [docs](https://github.com/kieraneglin/pinchflat/wiki/Prometheus-and-Grafana) |
+
+### Reverse Proxies
+
+Pinchflat makes heavy use of websockets for real-time updates. If you're running Pinchflat behind a reverse proxy then you'll need to make sure it's configured to support websockets.
 
 ## EFF donations
 

assets/tailwind.config.js

@@ -347,6 +347,38 @@ module.exports = {
         },
         { values }
       )
+    }),
+    plugin(function ({ matchComponents, theme }) {
+      let iconsDir = path.join(__dirname, './vendor/simple-icons')
+      let values = {}
+
+      fs.readdirSync(iconsDir).forEach((file) => {
+        let name = path.basename(file, '.svg')
+        values[name] = { name, fullPath: path.join(iconsDir, file) }
+      })
+
+      matchComponents(
+        {
+          si: ({ name, fullPath }) => {
+            let content = fs
+              .readFileSync(fullPath)
+              .toString()
+              .replace(/\r?\n|\r/g, '')
+            return {
+              [`--si-${name}`]: `url('data:image/svg+xml;utf8,${content}')`,
+              '-webkit-mask': `var(--si-${name})`,
+              mask: `var(--si-${name})`,
+              'mask-repeat': 'no-repeat',
+              'background-color': 'currentColor',
+              'vertical-align': 'middle',
+              display: 'inline-block',
+              width: theme('spacing.5'),
+              height: theme('spacing.5')
+            }
+          }
+        },
+        { values }
+      )
     })
   ]
 }

assets/vendor/simple-icons/discord.svg

@@ -0,0 +1 @@
+<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><title>Discord</title><path d="M20.317 4.3698a19.7913 19.7913 0 00-4.8851-1.5152.0741.0741 0 00-.0785.0371c-.211.3753-.4447.8648-.6083 1.2495-1.8447-.2762-3.68-.2762-5.4868 0-.1636-.3933-.4058-.8742-.6177-1.2495a.077.077 0 00-.0785-.037 19.7363 19.7363 0 00-4.8852 1.515.0699.0699 0 00-.0321.0277C.5334 9.0458-.319 13.5799.0992 18.0578a.0824.0824 0 00.0312.0561c2.0528 1.5076 4.0413 2.4228 5.9929 3.0294a.0777.0777 0 00.0842-.0276c.4616-.6304.8731-1.2952 1.226-1.9942a.076.076 0 00-.0416-.1057c-.6528-.2476-1.2743-.5495-1.8722-.8923a.077.077 0 01-.0076-.1277c.1258-.0943.2517-.1923.3718-.2914a.0743.0743 0 01.0776-.0105c3.9278 1.7933 8.18 1.7933 12.0614 0a.0739.0739 0 01.0785.0095c.1202.099.246.1981.3728.2924a.077.077 0 01-.0066.1276 12.2986 12.2986 0 01-1.873.8914.0766.0766 0 00-.0407.1067c.3604.698.7719 1.3628 1.225 1.9932a.076.076 0 00.0842.0286c1.961-.6067 3.9495-1.5219 6.0023-3.0294a.077.077 0 00.0313-.0552c.5004-5.177-.8382-9.6739-3.5485-13.6604a.061.061 0 00-.0312-.0286zM8.02 15.3312c-1.1825 0-2.1569-1.0857-2.1569-2.419 0-1.3332.9555-2.4189 2.157-2.4189 1.2108 0 2.1757 1.0952 2.1568 2.419 0 1.3332-.9555 2.4189-2.1569 2.4189zm7.9748 0c-1.1825 0-2.1569-1.0857-2.1569-2.419 0-1.3332.9554-2.4189 2.1569-2.4189 1.2108 0 2.1757 1.0952 2.1568 2.419 0 1.3332-.946 2.4189-2.1568 2.4189Z"/></svg>

assets/vendor/simple-icons/github.svg

@@ -0,0 +1 @@
+<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><title>GitHub</title><path d="M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12"/></svg>

config/config.exs

@@ -99,6 +99,12 @@ config :logger, :default_formatter,
 # Use Jason for JSON parsing in Phoenix
 config :phoenix, :json_library, Jason
 
+config :pinchflat, Pinchflat.PromEx,
+  disabled: true,
+  manual_metrics_start_delay: :no_delay,
+  drop_metrics_groups: [],
+  metrics_server: :disabled
+
 # Import environment specific config. This must remain at the bottom
 # of this file so it overrides the configuration defined above.
 import_config "#{config_env()}.exs"

config/dev.exs

@@ -81,3 +81,5 @@ config :phoenix_live_view, :debug_heex_annotations, true
 
 # Disable swoosh api client as it is only required for production adapters.
 config :swoosh, :api_client, false
+
+config :pinchflat, Pinchflat.PromEx, disabled: false

config/runtime.exs

@@ -72,6 +72,7 @@ if config_env() == :prod do
   # For running PF in a subdirectory via a reverse proxy
   base_route_path = System.get_env("BASE_ROUTE_PATH", "/")
   enable_ipv6 = String.length(System.get_env("ENABLE_IPV6", "")) > 0
+  enable_prometheus = String.length(System.get_env("ENABLE_PROMETHEUS", "")) > 0
 
   config :logger, level: String.to_existing_atom(System.get_env("LOG_LEVEL", "debug"))
 
@@ -95,6 +96,8 @@ if config_env() == :prod do
     database: db_path,
     journal_mode: journal_mode
 
+  config :pinchflat, Pinchflat.PromEx, disabled: !enable_prometheus
+
   # The secret key base is used to sign/encrypt cookies and other secrets.
   # A default value is used in config/dev.exs and config/test.exs but you
   # want to use a different value for prod and you most likely don't want

docker/selfhosted.Dockerfile

@@ -129,7 +129,7 @@ EXPOSE ${PORT}
 # Only copy the final release from the build stage
 COPY --from=builder /app/_build/${MIX_ENV}/rel/pinchflat ./
 
-HEALTHCHECK --interval=120s --start-period=10s \
+HEALTHCHECK --interval=30s --start-period=15s \
   CMD curl --fail http://localhost:${PORT}/healthcheck || exit 1
 
 # Start the app

lib/pinchflat/application.ex

@@ -11,6 +11,7 @@ defmodule Pinchflat.Application do
     check_and_update_timezone()
 
     children = [
+      Pinchflat.PromEx,
       PinchflatWeb.Telemetry,
       Pinchflat.Repo,
       # Must be before startup tasks

lib/pinchflat/downloading/download_option_builder.ex

@@ -152,6 +152,7 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
     case {behaviour, categories} do
       {_, []} -> []
       {:remove, _} -> [sponsorblock_remove: Enum.join(categories, ",")]
+      {:mark, _} -> [sponsorblock_mark: Enum.join(categories, ",")]
       {:disabled, _} -> []
     end
   end
@@ -200,6 +201,9 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilder do
     source = media_item_with_preloads.source
 
     %{
+      "media_item_id" => to_string(media_item_with_preloads.id),
+      "source_id" => to_string(source.id),
+      "media_profile_id" => to_string(source.media_profile_id),
       "source_custom_name" => source.custom_name,
       "source_collection_id" => source.collection_id,
       "source_collection_name" => source.collection_name,

lib/pinchflat/downloading/media_download_worker.ex

@@ -115,7 +115,7 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
   defp action_on_error(message) do
     # This will attempt re-download at the next indexing, but it won't be retried
     # immediately as part of job failure logic
-    non_retryable_errors = ["Video unavailable"]
+    non_retryable_errors = ["Video unavailable", "Sign in to confirm"]
 
     if String.contains?(to_string(message), non_retryable_errors) do
       Logger.error("yt-dlp download will not be retried: #{inspect(message)}")

lib/pinchflat/media/media_item.ex

@@ -112,6 +112,9 @@ defmodule Pinchflat.Media.MediaItem do
     |> dynamic_default(:uuid, fn _ -> Ecto.UUID.generate() end)
     |> update_upload_date_index()
     |> validate_required(@required_fields)
+    # Validate that the title does NOT start with "youtube video #" since that indicates a restriction by YouTube.
+    # See issue #549 for more information.
+    |> validate_format(:title, ~r/^(?!youtube video #)/)
     |> unique_constraint([:media_id, :source_id])
   end
 

lib/pinchflat/profiles/media_profile.ex

@@ -55,7 +55,7 @@ defmodule Pinchflat.Profiles.MediaProfile do
     field :embed_metadata, :boolean, default: false
 
     field :download_nfo, :boolean, default: false
-    field :sponsorblock_behaviour, Ecto.Enum, values: [:disabled, :remove], default: :disabled
+    field :sponsorblock_behaviour, Ecto.Enum, values: [:disabled, :mark, :remove], default: :disabled
     field :sponsorblock_categories, {:array, :string}, default: []
     # NOTE: these do NOT speed up indexing - the indexer still has to go
     # through the entire collection to determine if a media is a short or

lib/pinchflat/prom_ex.ex

@@ -0,0 +1,40 @@
+defmodule Pinchflat.PromEx do
+  @moduledoc """
+  Configuration for the PromEx library which provides Prometheus metrics
+  """
+
+  use PromEx, otp_app: :pinchflat
+
+  alias PromEx.Plugins
+
+  @impl true
+  def plugins do
+    [
+      Plugins.Application,
+      Plugins.Beam,
+      {Plugins.Phoenix, router: PinchflatWeb.Router, endpoint: PinchflatWeb.Endpoint},
+      Plugins.Ecto,
+      Plugins.Oban,
+      Plugins.PhoenixLiveView
+    ]
+  end
+
+  @impl true
+  def dashboard_assigns do
+    [
+      default_selected_interval: "30s"
+    ]
+  end
+
+  @impl true
+  def dashboards do
+    [
+      {:prom_ex, "application.json"},
+      {:prom_ex, "beam.json"},
+      {:prom_ex, "phoenix.json"},
+      {:prom_ex, "ecto.json"},
+      {:prom_ex, "oban.json"},
+      {:prom_ex, "phoenix_live_view.json"}
+    ]
+  end
+end

lib/pinchflat/settings/setting.ex

@@ -14,15 +14,17 @@ defmodule Pinchflat.Settings.Setting do
     :apprise_server,
     :video_codec_preference,
     :audio_codec_preference,
-    :youtube_api_key
+    :youtube_api_key,
+    :extractor_sleep_interval_seconds
   ]
 
-  @required_fields ~w(
+  @required_fields [
-    onboarding
+    :onboarding,
-    pro_enabled
+    :pro_enabled,
-    video_codec_preference
+    :video_codec_preference,
-    audio_codec_preference
+    :audio_codec_preference,
-  )a
+    :extractor_sleep_interval_seconds
+  ]
 
   schema "settings" do
     field :onboarding, :boolean, default: true
@@ -32,6 +34,7 @@ defmodule Pinchflat.Settings.Setting do
     field :apprise_server, :string
     field :youtube_api_key, :string
     field :route_token, :string
+    field :extractor_sleep_interval_seconds, :integer, default: 0
 
     field :video_codec_preference, :string
     field :audio_codec_preference, :string
@@ -42,5 +45,6 @@ defmodule Pinchflat.Settings.Setting do
     setting
     |> cast(attrs, @allowed_fields)
     |> validate_required(@required_fields)
+    |> validate_number(:extractor_sleep_interval_seconds, greater_than: 0)
   end
 end

lib/pinchflat/slow_indexing/media_collection_indexing_worker.ex

@@ -79,21 +79,21 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorker do
     case {source.index_frequency_minutes, source.last_indexed_at} do
       {index_freq, _} when index_freq > 0 ->
         # If the indexing is on a schedule simply run indexing and reschedule
-        perform_indexing_and_notification(source)
+        perform_indexing_and_notification(source, was_forced: args["force"])
         maybe_enqueue_fast_indexing_task(source)
         reschedule_indexing(source)
 
       {_, nil} ->
         # If the source has never been indexed, index it once
         # even if it's not meant to reschedule
-        perform_indexing_and_notification(source)
+        perform_indexing_and_notification(source, was_forced: args["force"])
         :ok
 
       _ ->
         # If the source HAS been indexed and is not meant to reschedule,
         # perform a no-op (unless forced)
         if args["force"] do
-          perform_indexing_and_notification(source)
+          perform_indexing_and_notification(source, was_forced: true)
         end
 
         :ok
@@ -103,11 +103,11 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorker do
     Ecto.StaleEntryError -> Logger.info("#{__MODULE__} discarded: source #{source_id} stale")
   end
 
-  defp perform_indexing_and_notification(source) do
+  defp perform_indexing_and_notification(source, indexing_opts) do
     apprise_server = Settings.get!(:apprise_server)
 
     SourceNotifications.wrap_new_media_notification(apprise_server, source, fn ->
-      SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
+      SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source, indexing_opts)
     end)
   end
 

lib/pinchflat/slow_indexing/slow_indexing_helpers.ex

@@ -5,6 +5,8 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
   Many of these methods are made to be kickoff or be consumed by workers.
   """
 
+  use Pinchflat.Media.MediaQuery
+
   require Logger
 
   alias Pinchflat.Repo
@@ -14,6 +16,7 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
   alias Pinchflat.Sources.Source
   alias Pinchflat.Media.MediaItem
   alias Pinchflat.YtDlp.MediaCollection
+  alias Pinchflat.Utils.FilesystemUtils
   alias Pinchflat.Downloading.DownloadingHelpers
   alias Pinchflat.SlowIndexing.FileFollowerServer
   alias Pinchflat.Downloading.DownloadOptionBuilder
@@ -22,13 +25,19 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
   alias Pinchflat.YtDlp.Media, as: YtDlpMedia
 
   @doc """
-  Starts tasks for indexing a source's media regardless of the source's indexing
+  Kills old indexing tasks and starts a new task to index the media collection.
-  frequency. It's assumed the caller will check for indexing frequency.
+
+  The job is delayed based on the source's `index_frequency_minutes` setting unless
+  one of the following is true:
+    - The `force` option is set to true
+    - The source has never been indexed before
+    - The source has been indexed before, but the last indexing job was more than
+      `index_frequency_minutes` ago
 
   Returns {:ok, %Task{}}
   """
   def kickoff_indexing_task(%Source{} = source, job_args \\ %{}, job_opts \\ []) do
-    job_offset_seconds = calculate_job_offset_seconds(source)
+    job_offset_seconds = if job_args[:force], do: 0, else: calculate_job_offset_seconds(source)
 
     Tasks.delete_pending_tasks_for(source, "FastIndexingWorker")
     Tasks.delete_pending_tasks_for(source, "MediaCollectionIndexingWorker", include_executing: true)
@@ -52,8 +61,8 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
   @doc """
   Given a media source, creates (indexes) the media by creating media_items for each
   media ID in the source. Afterward, kicks off a download task for each pending media
-  item belonging to the source. You can't tell me the method name isn't descriptive!
+  item belonging to the source. Returns a list of media items or changesets
-  Returns a list of media items or changesets (if the media item couldn't be created).
+  (if the media item couldn't be created).
 
   Indexing is slow and usually returns a list of all media data at once for record creation.
   To help with this, we use a file follower to watch the file that yt-dlp writes to
@@ -61,23 +70,33 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
   clarity to the user experience. This has a few things to be aware of which are documented
   below in the file watcher setup method.
 
+  Additionally, in the case of a repeat index we create a download archive file that
+  contains some media IDs that we've indexed in the past. Note that this archive doesn't
+  contain the most recent IDs but rather a subset of IDs that are offset by some amount.
+  Practically, this means that we'll re-index a small handful of media that we've recently
+  indexed, but this is a good thing since it'll let us pick up on any recent changes to the
+  most recent media items.
+
+  We don't create a download archive for playlists (only channels), nor do we create one if
+  the indexing was forced by the user.
+
   NOTE: downloads are only enqueued if the source is set to download media. Downloads are
   also enqueued for ALL pending media items, not just the ones that were indexed in this
   job run. This should ensure that any stragglers are caught if, for some reason, they
   weren't enqueued or somehow got de-queued.
 
-  Since indexing returns all media data EVERY TIME, we that that opportunity to update
+  Available options:
-  indexing metadata for media items that have already been created.
+    - `was_forced`: Whether the indexing was forced by the user
 
   Returns [%MediaItem{} | %Ecto.Changeset{}]
   """
-  def index_and_enqueue_download_for_media_items(%Source{} = source) do
+  def index_and_enqueue_download_for_media_items(%Source{} = source, opts \\ []) do
     # The media_profile is needed to determine the quality options to _then_ determine a more
     # accurate predicted filepath
     source = Repo.preload(source, [:media_profile])
     # See the method definition below for more info on how file watchers work
     # (important reading if you're not familiar with it)
-    {:ok, media_attributes} = setup_file_watcher_and_kickoff_indexing(source)
+    {:ok, media_attributes} = setup_file_watcher_and_kickoff_indexing(source, opts)
     # Reload because the source may have been updated during the (long-running) indexing process
     # and important settings like `download_media` may have changed.
     source = Repo.reload!(source)
@@ -109,14 +128,16 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
   # It attempts a graceful shutdown of the file follower after the indexing is done,
   # but the FileFollowerServer will also stop itself if it doesn't see any activity
   # for a sufficiently long time.
-  defp setup_file_watcher_and_kickoff_indexing(source) do
+  defp setup_file_watcher_and_kickoff_indexing(source, opts) do
+    was_forced = Keyword.get(opts, :was_forced, false)
     {:ok, pid} = FileFollowerServer.start_link()
 
     handler = fn filepath -> setup_file_follower_watcher(pid, filepath, source) end
 
     command_opts =
       [output: DownloadOptionBuilder.build_output_path_for(source)] ++
-        DownloadOptionBuilder.build_quality_options_for(source)
+        DownloadOptionBuilder.build_quality_options_for(source) ++
+        build_download_archive_options(source, was_forced)
 
     runner_opts = [file_listener_handler: handler, use_cookies: source.use_cookies]
     result = MediaCollection.get_media_attributes_for_collection(source.original_url, command_opts, runner_opts)
@@ -166,4 +187,57 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpers do
 
     max(0, index_frequency_seconds - offset_seconds)
   end
+
+  # The download archive file works in tandem with --break-on-existing to stop
+  # yt-dlp once we've hit media items we've already indexed. But we generate
+  # this list with a bit of an offset so we do intentionally re-scan some media
+  # items to pick up any recent changes (see `get_media_items_for_download_archive`).
+  #
+  # From there, we format the media IDs in the way that yt-dlp expects (ie: "<extractor> <media_id>")
+  # and return the filepath to the caller.
+  defp create_download_archive_file(source) do
+    tmpfile = FilesystemUtils.generate_metadata_tmpfile(:txt)
+
+    archive_contents =
+      source
+      |> get_media_items_for_download_archive()
+      |> Enum.map_join("\n", fn media_item -> "youtube #{media_item.media_id}" end)
+
+    case File.write(tmpfile, archive_contents) do
+      :ok -> tmpfile
+      err -> err
+    end
+  end
+
+  # Sorting by `uploaded_at` is important because we want to re-index the most recent
+  # media items first but there is no guarantee of any correlation between ID and uploaded_at.
+  #
+  # The offset is important because we want to re-index some media items that we've
+  # recently indexed to pick up on any changes. The limit is because we want this mechanism
+  # to work even if, for example, the video we were using as a stopping point was deleted.
+  # It's not a perfect system, but it should do well enough.
+  #
+  # The chosen limit and offset are arbitary, independent, and vibes-based. Feel free to
+  # tweak as-needed
+  defp get_media_items_for_download_archive(source) do
+    MediaQuery.new()
+    |> where(^MediaQuery.for_source(source))
+    |> order_by(desc: :uploaded_at)
+    |> limit(50)
+    |> offset(20)
+    |> Repo.all()
+  end
+
+  # The download archive isn't useful for playlists (since those are ordered arbitrarily)
+  # and we don't want to use it if the indexing was forced by the user. In other words,
+  # only create an archive for channels that are being indexed as part of their regular
+  # indexing schedule
+  defp build_download_archive_options(%Source{collection_type: :playlist}, _was_forced), do: []
+  defp build_download_archive_options(_source, true), do: []
+
+  defp build_download_archive_options(source, _was_forced) do
+    archive_file = create_download_archive_file(source)
+
+    [:break_on_existing, download_archive: archive_file]
+  end
 end

lib/pinchflat/sources/sources.ex

@@ -180,9 +180,12 @@ defmodule Pinchflat.Sources do
   end
 
   defp add_source_details_to_changeset(source, changeset) do
+    original_url = changeset.changes.original_url
     use_cookies = Ecto.Changeset.get_field(changeset, :use_cookies)
+    # Skipping sleep interval since this is UI blocking and we want to keep this as fast as possible
+    addl_opts = [use_cookies: use_cookies, skip_sleep_interval: true]
 
-    case MediaCollection.get_source_details(changeset.changes.original_url, [], use_cookies: use_cookies) do
+    case MediaCollection.get_source_details(original_url, [], addl_opts) do
       {:ok, source_details} ->
         add_source_details_by_collection_type(source, changeset, source_details)
 

lib/pinchflat/utils/filesystem_utils.ex

@@ -45,8 +45,20 @@ defmodule Pinchflat.Utils.FilesystemUtils do
   Returns binary()
   """
   def generate_metadata_tmpfile(type) do
+    filename = StringUtils.random_string(64)
+    # This "namespacing" is more to help with development since things get
+    # weird in my editor when there are thousands of files in a single directory
+    first_two = String.slice(filename, 0..1)
+    second_two = String.slice(filename, 2..3)
     tmpfile_directory = Application.get_env(:pinchflat, :tmpfile_directory)
-    filepath = Path.join([tmpfile_directory, "#{StringUtils.random_string(64)}.#{type}"])
+
+    filepath =
+      Path.join([
+        tmpfile_directory,
+        first_two,
+        second_two,
+        "#{filename}.#{type}"
+      ])
 
     :ok = write_p!(filepath, "")
 

lib/pinchflat/utils/number_utils.ex

@@ -36,4 +36,18 @@ defmodule Pinchflat.Utils.NumberUtils do
       end
     end)
   end
+
+  @doc """
+  Adds jitter to a number based on a percentage. Returns 0 if the number is less than or equal to 0.
+
+  Returns integer()
+  """
+  def add_jitter(num, jitter_percentage \\ 0.5)
+  def add_jitter(num, _jitter_percentage) when num <= 0, do: 0
+
+  def add_jitter(num, jitter_percentage) do
+    jitter = :rand.uniform(round(num * jitter_percentage))
+
+    round(num + jitter)
+  end
 end

lib/pinchflat/yt_dlp/command_runner.ex

@@ -5,7 +5,9 @@ defmodule Pinchflat.YtDlp.CommandRunner do
 
   require Logger
 
+  alias Pinchflat.Settings
   alias Pinchflat.Utils.CliUtils
+  alias Pinchflat.Utils.NumberUtils
   alias Pinchflat.YtDlp.YtDlpCommandRunner
   alias Pinchflat.Utils.FilesystemUtils, as: FSUtils
 
@@ -22,24 +24,30 @@ defmodule Pinchflat.YtDlp.CommandRunner do
       for a file watcher.
     - :use_cookies - if true, will add a cookie file to the command options. Will not
       attach a cookie file if the user hasn't set one up.
+    - :skip_sleep_interval - if true, will not add the sleep interval options to the command.
+      Usually only used for commands that would be UI-blocking
 
   Returns {:ok, binary()} | {:error, output, status}.
   """
   @impl YtDlpCommandRunner
   def run(url, action_name, command_opts, output_template, addl_opts \\ []) do
     Logger.debug("Running yt-dlp command for action: #{action_name}")
-    # This approach lets us mock the command for testing
-    command = backend_executable()
 
     output_filepath = generate_output_filepath(addl_opts)
     print_to_file_opts = [{:print_to_file, output_template}, output_filepath]
-    user_configured_opts = cookie_file_options(addl_opts)
+    user_configured_opts = cookie_file_options(addl_opts) ++ sleep_interval_opts(addl_opts)
     # These must stay in exactly this order, hence why I'm giving it its own variable.
     all_opts = command_opts ++ print_to_file_opts ++ user_configured_opts ++ global_options()
     formatted_command_opts = [url] ++ CliUtils.parse_options(all_opts)
 
-    case CliUtils.wrap_cmd(command, formatted_command_opts, stderr_to_stdout: true) do
+    case CliUtils.wrap_cmd(backend_executable(), formatted_command_opts, stderr_to_stdout: true) do
-      {_, 0} ->
+      # yt-dlp exit codes:
+      #   0 = Everything is successful
+      #   100 = yt-dlp must restart for update to complete
+      #   101 = Download cancelled by --max-downloads etc
+      #     2 = Error in user-provided options
+      #     1 = Any other error
+      {_, status} when status in [0, 101] ->
         # IDEA: consider deleting the file after reading it. It's in the tmp dir, so it's not
         # a huge deal, but it's still a good idea to clean up after ourselves.
         # (even on error? especially on error?)
@@ -90,6 +98,20 @@ defmodule Pinchflat.YtDlp.CommandRunner do
     end
   end
 
+  defp sleep_interval_opts(addl_opts) do
+    sleep_interval = Settings.get!(:extractor_sleep_interval_seconds)
+
+    if sleep_interval <= 0 || Keyword.get(addl_opts, :skip_sleep_interval) do
+      []
+    else
+      [
+        sleep_requests: NumberUtils.add_jitter(sleep_interval),
+        sleep_interval: NumberUtils.add_jitter(sleep_interval),
+        sleep_subtitles: NumberUtils.add_jitter(sleep_interval)
+      ]
+    end
+  end
+
   defp add_cookie_file do
     base_dir = Application.get_env(:pinchflat, :extras_directory)
     filename_options_map = %{cookies: "cookies.txt"}

lib/pinchflat/yt_dlp/media_collection.ex

@@ -23,7 +23,6 @@ defmodule Pinchflat.YtDlp.MediaCollection do
   Returns {:ok, [map()]} | {:error, any, ...}.
   """
   def get_media_attributes_for_collection(url, command_opts \\ [], addl_opts \\ []) do
-    runner = Application.get_env(:pinchflat, :yt_dlp_runner)
     # `ignore_no_formats_error` is necessary because yt-dlp will error out if
     # the first video has not released yet (ie: is a premier). We don't care about
     # available formats since we're just getting the media details
@@ -39,7 +38,7 @@ defmodule Pinchflat.YtDlp.MediaCollection do
       file_listener_handler.(output_filepath)
     end
 
-    case runner.run(url, action, all_command_opts, output_template, runner_opts) do
+    case backend_runner().run(url, action, all_command_opts, output_template, runner_opts) do
       {:ok, output} ->
         parsed_lines =
           output

lib/pinchflat_web/components/core_components.ex

@@ -700,7 +700,7 @@ defmodule PinchflatWeb.CoreComponents do
   attr :class, :string, default: nil
   attr :rest, :global
 
-  def icon(%{name: "hero-" <> _} = assigns) do
+  def icon(assigns) do
     ~H"""
     <span class={[@name, @class]} {@rest} />
     """

lib/pinchflat_web/components/layouts.ex

@@ -15,11 +15,12 @@ defmodule PinchflatWeb.Layouts do
   attr :text, :string, required: true
   attr :href, :any, required: true
   attr :target, :any, default: "_self"
+  attr :icon_class, :string, default: ""
 
   def sidebar_item(assigns) do
     ~H"""
     <li class="text-bodydark1">
-      <.sidebar_link icon={@icon} text={@text} href={@href} target={@target} />
+      <.sidebar_link icon={@icon} text={@text} href={@href} target={@target} icon_class={@icon_class} />
     </li>
     """
   end
@@ -89,6 +90,7 @@ defmodule PinchflatWeb.Layouts do
   attr :href, :any, required: true
   attr :target, :any, default: "_self"
   attr :class, :string, default: ""
+  attr :icon_class, :string, default: ""
 
   def sidebar_link(assigns) do
     ~H"""
@@ -103,7 +105,7 @@ defmodule PinchflatWeb.Layouts do
         @class
       ]}
     >
-      <.icon :if={@icon} name={@icon} /> {@text}
+      <.icon :if={@icon} name={@icon} class={@icon_class} /> {@text}
     </.link>
     """
   end

lib/pinchflat_web/components/layouts/partials/sidebar.html.heex

@@ -47,8 +47,10 @@
           text="Docs"
           target="_blank"
           href="https://github.com/kieraneglin/pinchflat/wiki"
+          icon_class="scale-110"
         />
-        <.sidebar_item icon="hero-cog" text="Github" target="_blank" href="https://github.com/kieraneglin/pinchflat" />
+        <.sidebar_item icon="si-github" text="Github" target="_blank" href="https://github.com/kieraneglin/pinchflat" />
+        <.sidebar_item icon="si-discord" text="Discord" target="_blank" href="https://discord.gg/j7T6dCuwU4" />
         <li>
           <span
             class={[
@@ -59,7 +61,7 @@
             ]}
             phx-click={show_modal("donate-modal")}
           >
-            <.icon name="hero-currency-dollar" /> Donate
+            <.icon name="hero-currency-dollar" class="scale-110" /> Donate
           </span>
         </li>
         <li>

lib/pinchflat_web/controllers/media_profiles/media_profile_html.ex

@@ -38,6 +38,7 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
   def friendly_sponsorblock_options do
     [
       {"Disabled (default)", "disabled"},
+      {"Mark Segments as Chapters", "mark"},
       {"Remove Segments", "remove"}
     ]
   end
@@ -56,7 +57,7 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
   end
 
   def media_center_custom_output_template_options do
-    %{
+    [
       season_by_year__episode_by_date: "<code>Season YYYY/sYYYYeMMDD</code>",
       season_by_year__episode_by_date_and_index:
         "same as the above but it handles dates better. <strong>This is the recommended option</strong>",
@@ -64,11 +65,11 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
         "<code>Season 1/s01eXX</code> where <code>XX</code> is the video's position in the playlist. Only recommended for playlists (not channels) that don't change",
       static_season__episode_by_date:
         "<code>Season 1/s01eYYMMDD</code>. Recommended for playlists that might change or where order isn't important"
-    }
+    ]
   end
 
   def other_custom_output_template_options do
-    %{
+    [
       upload_day: nil,
       upload_month: nil,
       upload_year: nil,
@@ -84,8 +85,11 @@ defmodule PinchflatWeb.MediaProfiles.MediaProfileHTML do
       season_episode_index_from_date:
         "the upload date formatted as <code>sYYYYeMMDDII</code> where <code>II</code> is an index to prevent date collisions",
       media_playlist_index:
-        "the place of the media item in the playlist. Do not use with channels. May not work if the playlist is updated"
+        "the place of the media item in the playlist. Do not use with channels. May not work if the playlist is updated",
-    }
+      media_item_id: "the ID of the media item in Pinchflat's database",
+      source_id: "the ID of the source in Pinchflat's database",
+      media_profile_id: "the ID of the media profile in Pinchflat's database"
+    ]
   end
 
   def common_output_template_options do

lib/pinchflat_web/controllers/settings/setting_html/setting_form.html.heex

@@ -29,7 +29,7 @@
   <section class="mt-8">
     <section>
       <h3 class="text-2xl text-black dark:text-white">
-        Indexing Settings
+        Extractor Settings
       </h3>
 
       <.input
@@ -41,6 +41,14 @@
         html_help={true}
         inputclass="font-mono text-sm mr-4"
       />
+
+      <.input
+        field={f[:extractor_sleep_interval_seconds]}
+        placeholder="0"
+        type="number"
+        label="Sleep Interval (seconds)"
+        help="Sleep interval in seconds between each extractor request. Must be a positive whole number (or set to 0 to disable)"
+      />
     </section>
   </section>
 

lib/pinchflat_web/controllers/sources/source_html/actions_dropdown.html.heex

@@ -54,7 +54,7 @@
     <.link
       href={~p"/sources/#{@source}/force_index"}
       method="post"
-      data-confirm="Are you sure you want to force an index of this source? This isn't normally needed."
+      data-confirm="Are you sure you want index all content from this source? This isn't normally needed."
     >
       Force Index
     </.link>

lib/pinchflat_web/endpoint.ex

@@ -32,6 +32,8 @@ defmodule PinchflatWeb.Endpoint do
     plug Phoenix.Ecto.CheckRepoStatus, otp_app: :pinchflat
   end
 
+  plug PromEx.Plug, prom_ex_module: Pinchflat.PromEx
+
   plug Phoenix.LiveDashboard.RequestLogger,
     param_key: "request_logger",
     cookie_key: "request_logger"

lib/pinchflat_web/router.ex

@@ -25,7 +25,6 @@ defmodule PinchflatWeb.Router do
 
     # has to match before /sources/:id
     get "/sources/opml", Podcasts.PodcastController, :opml_feed
-    get "/sources/:foo/opml", Podcasts.PodcastController, :opml_feed
   end
 
   # Routes in here _may not be_ protected by basic auth. This is necessary for

mix.exs

@@ -4,7 +4,7 @@ defmodule Pinchflat.MixProject do
   def project do
     [
       app: :pinchflat,
-      version: "2024.12.31",
+      version: "2025.1.14",
       elixir: "~> 1.17",
       elixirc_paths: elixirc_paths(Mix.env()),
       start_permanent: Mix.env() == :prod,
@@ -60,9 +60,9 @@ defmodule Pinchflat.MixProject do
       {:esbuild, "~> 0.8", runtime: Mix.env() == :dev},
       {:tailwind, "~> 0.2.0", runtime: Mix.env() == :dev},
       {:swoosh, "~> 1.3"},
-      {:finch, "~> 0.13"},
+      {:finch, "~> 0.18"},
-      {:telemetry_metrics, "~> 0.6"},
+      {:telemetry_metrics, "~> 1.0"},
-      {:telemetry_poller, "~> 1.0"},
+      {:telemetry_poller, "~> 1.1"},
       {:gettext, "~> 0.20"},
       {:jason, "~> 1.2"},
       {:dns_cluster, "~> 0.1.1"},
@@ -70,6 +70,7 @@ defmodule Pinchflat.MixProject do
       {:oban, "~> 2.17"},
       {:nimble_parsec, "~> 1.4"},
       {:timex, "~> 3.0"},
+      {:prom_ex, "~> 1.11.0"},
       {:mox, "~> 1.0", only: :test},
       {:credo, "~> 1.7.7", only: [:dev, :test], runtime: false},
       {:credo_naming, "~> 2.1", only: [:dev, :test], runtime: false},

mix.lock

@@ -1,6 +1,6 @@
 %{
   "bunt": {:hex, :bunt, "1.0.0", "081c2c665f086849e6d57900292b3a161727ab40431219529f13c4ddcf3e7a44", [:mix], [], "hexpm", "dc5f86aa08a5f6fa6b8096f0735c4e76d54ae5c9fa2c143e5a1fc7c1cd9bb6b5"},
-  "castore": {:hex, :castore, "1.0.10", "43bbeeac820f16c89f79721af1b3e092399b3a1ecc8df1a472738fd853574911", [:mix], [], "hexpm", "1b0b7ea14d889d9ea21202c43a4fa015eb913021cb535e8ed91946f4b77a8848"},
+  "castore": {:hex, :castore, "1.0.11", "4bbd584741601eb658007339ea730b082cc61f3554cf2e8f39bf693a11b49073", [:mix], [], "hexpm", "e03990b4db988df56262852f20de0f659871c35154691427a5047f4967a16a62"},
   "cc_precompiler": {:hex, :cc_precompiler, "0.1.10", "47c9c08d8869cf09b41da36538f62bc1abd3e19e41701c2cea2675b53c704258", [:mix], [{:elixir_make, "~> 0.7", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "f6e046254e53cd6b41c6bacd70ae728011aa82b2742a80d6e2214855c6e06b22"},
   "certifi": {:hex, :certifi, "2.12.0", "2d1cca2ec95f59643862af91f001478c9863c2ac9cb6e2f89780bfd8de987329", [:rebar3], [], "hexpm", "ee68d85df22e554040cdb4be100f33873ac6051387baf6a8f6ce82272340ff1c"},
   "combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm", "1b1dbc1790073076580d0d1d64e42eae2366583e7aecd455d1215b0d16f2451b"},
@@ -23,23 +23,25 @@
   "exqlite": {:hex, :exqlite, "0.23.0", "6e851c937a033299d0784994c66da24845415072adbc455a337e20087bce9033", [:make, :mix], [{:cc_precompiler, "~> 0.1", [hex: :cc_precompiler, repo: "hexpm", optional: false]}, {:db_connection, "~> 2.1", [hex: :db_connection, repo: "hexpm", optional: false]}, {:elixir_make, "~> 0.8", [hex: :elixir_make, repo: "hexpm", optional: false]}, {:table, "~> 0.1.0", [hex: :table, repo: "hexpm", optional: true]}], "hexpm", "404341cceec5e6466aaed160cf0b58be2019b60af82588c215e1224ebd3ec831"},
   "faker": {:hex, :faker, "0.17.0", "671019d0652f63aefd8723b72167ecdb284baf7d47ad3a82a15e9b8a6df5d1fa", [:mix], [], "hexpm", "a7d4ad84a93fd25c5f5303510753789fc2433ff241bf3b4144d3f6f291658a6a"},
   "file_system": {:hex, :file_system, "0.2.10", "fb082005a9cd1711c05b5248710f8826b02d7d1784e7c3451f9c1231d4fc162d", [:mix], [], "hexpm", "41195edbfb562a593726eda3b3e8b103a309b733ad25f3d642ba49696bf715dc"},
-  "finch": {:hex, :finch, "0.17.0", "17d06e1d44d891d20dbd437335eebe844e2426a0cd7e3a3e220b461127c73f70", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: false]}, {:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 1.3", [hex: :mint, repo: "hexpm", optional: false]}, {:nimble_options, "~> 0.4 or ~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_pool, "~> 0.2.6 or ~> 1.0", [hex: :nimble_pool, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "8d014a661bb6a437263d4b5abf0bcbd3cf0deb26b1e8596f2a271d22e48934c7"},
+  "finch": {:hex, :finch, "0.19.0", "c644641491ea854fc5c1bbaef36bfc764e3f08e7185e1f084e35e0672241b76d", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 1.6.2 or ~> 1.7", [hex: :mint, repo: "hexpm", optional: false]}, {:nimble_options, "~> 0.4 or ~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_pool, "~> 1.1", [hex: :nimble_pool, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "fc5324ce209125d1e2fa0fcd2634601c52a787aff1cd33ee833664a5af4ea2b6"},
   "floki": {:hex, :floki, "0.36.3", "1102f93b16a55bc5383b85ae3ec470f82dee056eaeff9195e8afdf0ef2a43c30", [:mix], [], "hexpm", "fe0158bff509e407735f6d40b3ee0d7deb47f3f3ee7c6c182ad28599f9f6b27a"},
   "gettext": {:hex, :gettext, "0.24.0", "6f4d90ac5f3111673cbefc4ebee96fe5f37a114861ab8c7b7d5b30a1108ce6d8", [:mix], [{:expo, "~> 0.5.1", [hex: :expo, repo: "hexpm", optional: false]}], "hexpm", "bdf75cdfcbe9e4622dd18e034b227d77dd17f0f133853a1c73b97b3d6c770e8b"},
   "hackney": {:hex, :hackney, "1.20.1", "8d97aec62ddddd757d128bfd1df6c5861093419f8f7a4223823537bad5d064e2", [:rebar3], [{:certifi, "~> 2.12.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~> 6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~> 1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~> 1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.4.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~> 1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "fe9094e5f1a2a2c0a7d10918fee36bfec0ec2a979994cff8cfe8058cd9af38e3"},
-  "hpax": {:hex, :hpax, "0.1.2", "09a75600d9d8bbd064cdd741f21fc06fc1f4cf3d0fcc335e5aa19be1a7235c84", [:mix], [], "hexpm", "2c87843d5a23f5f16748ebe77969880e29809580efdaccd615cd3bed628a8c13"},
+  "hpax": {:hex, :hpax, "1.0.2", "762df951b0c399ff67cc57c3995ec3cf46d696e41f0bba17da0518d94acd4aac", [:mix], [], "hexpm", "2f09b4c1074e0abd846747329eaa26d535be0eb3d189fa69d812bfb8bfefd32f"},
   "idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"},
   "jason": {:hex, :jason, "1.4.4", "b9226785a9aa77b6857ca22832cffa5d5011a667207eb2a0ad56adb5db443b8a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "c5eb0cab91f094599f94d55bc63409236a8ec69a21a67814529e8d5f6cc90b3b"},
   "metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], [], "hexpm", "69b09adddc4f74a40716ae54d140f93beb0fb8978d8636eaded0c31b6f099f16"},
   "mime": {:hex, :mime, "2.0.6", "8f18486773d9b15f95f4f4f1e39b710045fa1de891fada4516559967276e4dc2", [:mix], [], "hexpm", "c9945363a6b26d747389aac3643f8e0e09d30499a138ad64fe8fd1d13d9b153e"},
   "mimerl": {:hex, :mimerl, "1.3.0", "d0cd9fc04b9061f82490f6581e0128379830e78535e017f7780f37fea7545726", [:rebar3], [], "hexpm", "a1e15a50d1887217de95f0b9b0793e32853f7c258a5cd227650889b38839fe9d"},
-  "mint": {:hex, :mint, "1.5.2", "4805e059f96028948870d23d7783613b7e6b0e2fb4e98d720383852a760067fd", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "d77d9e9ce4eb35941907f1d3df38d8f750c357865353e21d335bdcdf6d892a02"},
+  "mint": {:hex, :mint, "1.6.2", "af6d97a4051eee4f05b5500671d47c3a67dac7386045d87a904126fd4bbcea2e", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1 or ~> 0.2.0 or ~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "5ee441dffc1892f1ae59127f74afe8fd82fda6587794278d924e4d90ea3d63f9"},
   "mox": {:hex, :mox, "1.1.0", "0f5e399649ce9ab7602f72e718305c0f9cdc351190f72844599545e4996af73c", [:mix], [], "hexpm", "d44474c50be02d5b72131070281a5d3895c0e7a95c780e90bc0cfe712f633a13"},
-  "nimble_options": {:hex, :nimble_options, "1.1.0", "3b31a57ede9cb1502071fade751ab0c7b8dbe75a9a4c2b5bbb0943a690b63172", [:mix], [], "hexpm", "8bbbb3941af3ca9acc7835f5655ea062111c9c27bcac53e004460dfd19008a99"},
+  "nimble_options": {:hex, :nimble_options, "1.1.1", "e3a492d54d85fc3fd7c5baf411d9d2852922f66e69476317787a7b2bb000a61b", [:mix], [], "hexpm", "821b2470ca9442c4b6984882fe9bb0389371b8ddec4d45a9504f00a66f650b44"},
   "nimble_parsec": {:hex, :nimble_parsec, "1.4.0", "51f9b613ea62cfa97b25ccc2c1b4216e81df970acd8e16e8d1bdc58fef21370d", [:mix], [], "hexpm", "9c565862810fb383e9838c1dd2d7d2c437b3d13b267414ba6af33e50d2d1cf28"},
-  "nimble_pool": {:hex, :nimble_pool, "1.0.0", "5eb82705d138f4dd4423f69ceb19ac667b3b492ae570c9f5c900bb3d2f50a847", [:mix], [], "hexpm", "80be3b882d2d351882256087078e1b1952a28bf98d0a287be87e4a24a710b67a"},
+  "nimble_pool": {:hex, :nimble_pool, "1.1.0", "bf9c29fbdcba3564a8b800d1eeb5a3c58f36e1e11d7b7fb2e084a643f645f06b", [:mix], [], "hexpm", "af2e4e6b34197db81f7aad230c1118eac993acc0dae6bc83bac0126d4ae0813a"},
   "oban": {:hex, :oban, "2.18.3", "1608c04f8856c108555c379f2f56bc0759149d35fa9d3b825cb8a6769f8ae926", [:mix], [{:ecto_sql, "~> 3.10", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:ecto_sqlite3, "~> 0.9", [hex: :ecto_sqlite3, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: false]}, {:postgrex, "~> 0.16", [hex: :postgrex, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "36ca6ca84ef6518f9c2c759ea88efd438a3c81d667ba23b02b062a0aa785475e"},
+  "octo_fetch": {:hex, :octo_fetch, "0.4.0", "074b5ecbc08be10b05b27e9db08bc20a3060142769436242702931c418695b19", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~> 1.1", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm", "cf8be6f40cd519d7000bb4e84adcf661c32e59369ca2827c4e20042eda7a7fc6"},
   "parse_trans": {:hex, :parse_trans, "3.4.1", "6e6aa8167cb44cc8f39441d05193be6e6f4e7c2946cb2759f015f8c56b76e5ff", [:rebar3], [], "hexpm", "620a406ce75dada827b82e453c19cf06776be266f5a67cff34e1ef2cbb60e49a"},
+  "peep": {:hex, :peep, "3.4.1", "0e5263710fa0b42675bd0a11fdcdd3ee4f484e319105b6ad9a576c91a5d3cb55", [:mix], [{:nimble_options, "~> 1.1", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:plug, "~> 1.16", [hex: :plug, repo: "hexpm", optional: true]}, {:telemetry_metrics, "~> 1.0", [hex: :telemetry_metrics, repo: "hexpm", optional: false]}], "hexpm", "7a9b8c1f17b8b9475efb27b7048afa4d89ab84ef33a3d1df13696c85c12cd632"},
   "phoenix": {:hex, :phoenix, "1.7.17", "2fcdceecc6fb90bec26fab008f96abbd0fd93bc9956ec7985e5892cf545152ca", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:phoenix_pubsub, "~> 2.1", [hex: :phoenix_pubsub, repo: "hexpm", optional: false]}, {:phoenix_template, "~> 1.0", [hex: :phoenix_template, repo: "hexpm", optional: false]}, {:phoenix_view, "~> 2.0", [hex: :phoenix_view, repo: "hexpm", optional: true]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}, {:plug_cowboy, "~> 2.7", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:plug_crypto, "~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:websock_adapter, "~> 0.5.3", [hex: :websock_adapter, repo: "hexpm", optional: false]}], "hexpm", "50e8ad537f3f7b0efb1509b2f75b5c918f697be6a45d48e49a30d3b7c0e464c9"},
   "phoenix_ecto": {:hex, :phoenix_ecto, "4.4.3", "86e9878f833829c3f66da03d75254c155d91d72a201eb56ae83482328dc7ca93", [:mix], [{:ecto, "~> 3.5", [hex: :ecto, repo: "hexpm", optional: false]}, {:phoenix_html, "~> 2.14.2 or ~> 3.0 or ~> 4.0", [hex: :phoenix_html, repo: "hexpm", optional: true]}, {:plug, "~> 1.9", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "d36c401206f3011fefd63d04e8ef626ec8791975d9d107f9a0817d426f61ac07"},
   "phoenix_html": {:hex, :phoenix_html, "3.3.4", "42a09fc443bbc1da37e372a5c8e6755d046f22b9b11343bf885067357da21cb3", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "0249d3abec3714aff3415e7ee3d9786cb325be3151e6c4b3021502c585bf53fb"},
@@ -51,6 +53,7 @@
   "plug": {:hex, :plug, "1.16.1", "40c74619c12f82736d2214557dedec2e9762029b2438d6d175c5074c933edc9d", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.1.1 or ~> 1.2 or ~> 2.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "a13ff6b9006b03d7e33874945b2755253841b238c34071ed85b0e86057f8cddc"},
   "plug_cowboy": {:hex, :plug_cowboy, "2.7.2", "fdadb973799ae691bf9ecad99125b16625b1c6039999da5fe544d99218e662e4", [:mix], [{:cowboy, "~> 2.7", [hex: :cowboy, repo: "hexpm", optional: false]}, {:cowboy_telemetry, "~> 0.3", [hex: :cowboy_telemetry, repo: "hexpm", optional: false]}, {:plug, "~> 1.14", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm", "245d8a11ee2306094840c000e8816f0cbed69a23fc0ac2bcf8d7835ae019bb2f"},
   "plug_crypto": {:hex, :plug_crypto, "2.1.0", "f44309c2b06d249c27c8d3f65cfe08158ade08418cf540fd4f72d4d6863abb7b", [:mix], [], "hexpm", "131216a4b030b8f8ce0f26038bc4421ae60e4bb95c5cf5395e1421437824c4fa"},
+  "prom_ex": {:hex, :prom_ex, "1.11.0", "1f6d67f2dead92224cb4f59beb3e4d319257c5728d9638b4a5e8ceb51a4f9c7e", [:mix], [{:absinthe, ">= 1.7.0", [hex: :absinthe, repo: "hexpm", optional: true]}, {:broadway, ">= 1.1.0", [hex: :broadway, repo: "hexpm", optional: true]}, {:ecto, ">= 3.11.0", [hex: :ecto, repo: "hexpm", optional: true]}, {:finch, "~> 0.18", [hex: :finch, repo: "hexpm", optional: false]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: false]}, {:oban, ">= 2.10.0", [hex: :oban, repo: "hexpm", optional: true]}, {:octo_fetch, "~> 0.4", [hex: :octo_fetch, repo: "hexpm", optional: false]}, {:peep, "~> 3.0", [hex: :peep, repo: "hexpm", optional: false]}, {:phoenix, ">= 1.7.0", [hex: :phoenix, repo: "hexpm", optional: true]}, {:phoenix_live_view, ">= 0.20.0", [hex: :phoenix_live_view, repo: "hexpm", optional: true]}, {:plug, ">= 1.16.0", [hex: :plug, repo: "hexpm", optional: true]}, {:plug_cowboy, ">= 2.6.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:telemetry, ">= 1.0.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:telemetry_metrics, "~> 1.0", [hex: :telemetry_metrics, repo: "hexpm", optional: false]}, {:telemetry_metrics_prometheus_core, "~> 1.2", [hex: :telemetry_metrics_prometheus_core, repo: "hexpm", optional: false]}, {:telemetry_poller, "~> 1.1", [hex: :telemetry_poller, repo: "hexpm", optional: false]}], "hexpm", "76b074bc3730f0802978a7eb5c7091a65473eaaf07e99ec9e933138dcc327805"},
   "ranch": {:hex, :ranch, "1.8.0", "8c7a100a139fd57f17327b6413e4167ac559fbc04ca7448e9be9057311597a1d", [:make, :rebar3], [], "hexpm", "49fbcfd3682fab1f5d109351b61257676da1a2fdbe295904176d5e521a2ddfe5"},
   "sobelow": {:hex, :sobelow, "0.13.0", "218afe9075904793f5c64b8837cc356e493d88fddde126a463839351870b8d1e", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "cd6e9026b85fc35d7529da14f95e85a078d9dd1907a9097b3ba6ac7ebbe34a0d"},
   "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"},
@@ -58,8 +61,9 @@
   "table_rex": {:hex, :table_rex, "4.0.0", "3c613a68ebdc6d4d1e731bc973c233500974ec3993c99fcdabb210407b90959b", [:mix], [], "hexpm", "c35c4d5612ca49ebb0344ea10387da4d2afe278387d4019e4d8111e815df8f55"},
   "tailwind": {:hex, :tailwind, "0.2.2", "9e27288b568ede1d88517e8c61259bc214a12d7eed271e102db4c93fcca9b2cd", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}], "hexpm", "ccfb5025179ea307f7f899d1bb3905cd0ac9f687ed77feebc8f67bdca78565c4"},
   "telemetry": {:hex, :telemetry, "1.3.0", "fedebbae410d715cf8e7062c96a1ef32ec22e764197f70cda73d82778d61e7a2", [:rebar3], [], "hexpm", "7015fc8919dbe63764f4b4b87a95b7c0996bd539e0d499be6ec9d7f3875b79e6"},
-  "telemetry_metrics": {:hex, :telemetry_metrics, "0.6.2", "2caabe9344ec17eafe5403304771c3539f3b6e2f7fb6a6f602558c825d0d0bfb", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "9b43db0dc33863930b9ef9d27137e78974756f5f198cae18409970ed6fa5b561"},
+  "telemetry_metrics": {:hex, :telemetry_metrics, "1.0.0", "29f5f84991ca98b8eb02fc208b2e6de7c95f8bb2294ef244a176675adc7775df", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "f23713b3847286a534e005126d4c959ebcca68ae9582118ce436b521d1d47d5d"},
-  "telemetry_poller": {:hex, :telemetry_poller, "1.0.0", "db91bb424e07f2bb6e73926fcafbfcbcb295f0193e0a00e825e589a0a47e8453", [:rebar3], [{:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "b3a24eafd66c3f42da30fc3ca7dda1e9d546c12250a2d60d7b81d264fbec4f6e"},
+  "telemetry_metrics_prometheus_core": {:hex, :telemetry_metrics_prometheus_core, "1.2.1", "c9755987d7b959b557084e6990990cb96a50d6482c683fb9622a63837f3cd3d8", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}, {:telemetry_metrics, "~> 0.6 or ~> 1.0", [hex: :telemetry_metrics, repo: "hexpm", optional: false]}], "hexpm", "5e2c599da4983c4f88a33e9571f1458bf98b0cf6ba930f1dc3a6e8cf45d5afb6"},
+  "telemetry_poller": {:hex, :telemetry_poller, "1.1.0", "58fa7c216257291caaf8d05678c8d01bd45f4bdbc1286838a28c4bb62ef32999", [:rebar3], [{:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "9eb9d9cbfd81cbd7cdd24682f8711b6e2b691289a0de6826e58452f28c103c8f"},
   "timex": {:hex, :timex, "3.7.11", "bb95cb4eb1d06e27346325de506bcc6c30f9c6dea40d1ebe390b262fad1862d1", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.20", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 1.1", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "8b9024f7efbabaf9bd7aa04f65cf8dcd7c9818ca5737677c7b76acbc6a94d1aa"},
   "tzdata": {:hex, :tzdata, "1.1.2", "45e5f1fcf8729525ec27c65e163be5b3d247ab1702581a94674e008413eef50b", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "cec7b286e608371602318c414f344941d5eb0375e14cfdab605cca2fe66cba8b"},
   "unicode_util_compat": {:hex, :unicode_util_compat, "0.7.0", "bc84380c9ab48177092f43ac89e4dfa2c6d62b40b8bd132b1059ecc7232f9a78", [:rebar3], [], "hexpm", "25eee6d67df61960cf6a794239566599b09e17e668d3700247bc498638152521"},

priv/grafana/application.json

@@ -0,0 +1,607 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": "-- Grafana --",
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      },
+      {
+        "datasource": "-- Grafana --",
+        "enable": true,
+        "hide": true,
+        "iconColor": "#73BF69",
+        "limit": 100,
+        "name": "PromEx service start",
+        "showIn": 0,
+        "tags": ["prom_ex", "pinchflat", "start"],
+        "type": "tags"
+      },
+      {
+        "datasource": "-- Grafana --",
+        "enable": true,
+        "hide": true,
+        "iconColor": "#FF9830",
+        "limit": 100,
+        "name": "PromEx service stop",
+        "showIn": 0,
+        "tags": ["prom_ex", "pinchflat", "stop"],
+        "type": "tags"
+      }
+    ]
+  },
+  "description": "All the data that is presented here is captured by the PromEx Application plugin (https://github.com/akoutmos/prom_ex/blob/master/lib/prom_ex/plugins/application.ex)",
+  "editable": false,
+  "gnetId": null,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [
+    {
+      "asDropdown": false,
+      "icon": "bolt",
+      "includeVars": false,
+      "keepTime": false,
+      "tags": [],
+      "targetBlank": true,
+      "title": "Sponsor PromEx",
+      "tooltip": "",
+      "type": "link",
+      "url": "https://github.com/sponsors/akoutmos"
+    },
+    {
+      "asDropdown": false,
+      "icon": "doc",
+      "includeVars": false,
+      "keepTime": false,
+      "tags": [],
+      "targetBlank": true,
+      "title": "Application Plugin Docs",
+      "tooltip": "",
+      "type": "link",
+      "url": "https://hexdocs.pm/prom_ex/PromEx.Plugins.Application.html"
+    }
+  ],
+  "panels": [
+    {
+      "datasource": "prometheus",
+      "description": "The amount of time that the application has been running.",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {},
+          "decimals": 1,
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "dtdurationms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 8,
+        "x": 0,
+        "y": 0
+      },
+      "id": 6,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": ["last"],
+          "fields": "",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "7.1.3",
+      "targets": [
+        {
+          "expr": "pinchflat_prom_ex_application_uptime_milliseconds_count{job=\"$job\", instance=\"$instance\"}",
+          "format": "table",
+          "instant": true,
+          "interval": "",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "timeFrom": null,
+      "timeShift": null,
+      "title": "Application Uptime",
+      "type": "stat"
+    },
+    {
+      "datasource": "prometheus",
+      "description": "The data is populated by the PromEx Application plugin and provides information regarding your application's dependencies.",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "align": "left",
+            "displayMode": "auto"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "Status"
+            },
+            "properties": [
+              {
+                "id": "custom.displayMode",
+                "value": "color-background"
+              },
+              {
+                "id": "mappings",
+                "value": [
+                  {
+                    "from": "",
+                    "id": 0,
+                    "text": "Started",
+                    "to": "",
+                    "type": 1,
+                    "value": "1"
+                  },
+                  {
+                    "from": "",
+                    "id": 1,
+                    "text": "Loaded",
+                    "to": "",
+                    "type": 1,
+                    "value": "0"
+                  }
+                ]
+              },
+              {
+                "id": "custom.align",
+                "value": "center"
+              },
+              {
+                "id": "custom.width",
+                "value": 202
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "Name"
+            },
+            "properties": [
+              {
+                "id": "custom.width",
+                "value": 349
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "Version"
+            },
+            "properties": [
+              {
+                "id": "custom.width",
+                "value": 187
+              }
+            ]
+          }
+        ]
+      },
+      "gridPos": {
+        "h": 36,
+        "w": 16,
+        "x": 8,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "showHeader": true,
+        "sortBy": [
+          {
+            "desc": false,
+            "displayName": "Name"
+          }
+        ]
+      },
+      "pluginVersion": "7.1.3",
+      "targets": [
+        {
+          "expr": "pinchflat_prom_ex_application_dependency_info{job=\"$job\", instance=\"$instance\"}",
+          "format": "table",
+          "instant": true,
+          "interval": "",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "timeFrom": null,
+      "timeShift": null,
+      "title": "Dependency Information",
+      "transformations": [
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true,
+              "__name__": true,
+              "instance": true,
+              "job": true,
+              "Value": true
+            },
+            "indexByName": {
+              "Time": 0,
+              "Value": 4,
+              "__name__": 1,
+              "instance": 2,
+              "job": 3,
+              "modules": 7,
+              "name": 5,
+              "version": 6
+            },
+            "renameByName": {
+              "Value": "Status",
+              "modules": "Number of Modules Loaded",
+              "name": "Name",
+              "version": "Version"
+            }
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": "prometheus",
+      "description": "The name of the primary application that is running.",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {},
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 8,
+        "x": 0,
+        "y": 6
+      },
+      "id": 11,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": ["last"],
+          "fields": "/^name$/",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "7.1.3",
+      "targets": [
+        {
+          "expr": "pinchflat_prom_ex_application_primary_info{job=\"$job\", instance=\"$instance\"}",
+          "format": "table",
+          "instant": true,
+          "interval": "",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "timeFrom": null,
+      "timeShift": null,
+      "title": "Application Name",
+      "type": "stat"
+    },
+    {
+      "datasource": "prometheus",
+      "description": "The Git SHA of the application.",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {},
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 8,
+        "x": 0,
+        "y": 12
+      },
+      "id": 10,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": ["last"],
+          "fields": "/^sha$/",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "7.1.3",
+      "targets": [
+        {
+          "expr": "pinchflat_prom_ex_application_git_sha_info{job=\"$job\", instance=\"$instance\"}",
+          "format": "table",
+          "instant": true,
+          "interval": "",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "timeFrom": null,
+      "timeShift": null,
+      "title": "Application Git SHA",
+      "type": "stat"
+    },
+    {
+      "datasource": "prometheus",
+      "description": "The author of the application's last Git commit.",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {},
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 8,
+        "x": 0,
+        "y": 18
+      },
+      "id": 12,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": ["last"],
+          "fields": "/^author$/",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "7.1.3",
+      "targets": [
+        {
+          "expr": "pinchflat_prom_ex_application_git_author_info{job=\"$job\", instance=\"$instance\"}",
+          "format": "table",
+          "instant": true,
+          "interval": "",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "timeFrom": null,
+      "timeShift": null,
+      "title": "Application Git Author",
+      "type": "stat"
+    },
+    {
+      "datasource": "prometheus",
+      "description": "The version of the primary application that is running.",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {},
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 8,
+        "x": 0,
+        "y": 24
+      },
+      "id": 7,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": ["last"],
+          "fields": "/^version$/",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "7.1.3",
+      "targets": [
+        {
+          "expr": "pinchflat_prom_ex_application_primary_info{job=\"$job\", instance=\"$instance\"}",
+          "format": "table",
+          "instant": true,
+          "interval": "",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "timeFrom": null,
+      "timeShift": null,
+      "title": "Application Version",
+      "type": "stat"
+    },
+    {
+      "datasource": "prometheus",
+      "description": "The number of modules loaded by the primary application that is running.",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {},
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 6,
+        "w": 8,
+        "x": 0,
+        "y": 30
+      },
+      "id": 9,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": ["last"],
+          "fields": "/^modules$/",
+          "values": false
+        },
+        "textMode": "auto"
+      },
+      "pluginVersion": "7.1.3",
+      "targets": [
+        {
+          "expr": "pinchflat_prom_ex_application_primary_info{job=\"$job\", instance=\"$instance\"}",
+          "format": "table",
+          "instant": true,
+          "interval": "",
+          "legendFormat": "",
+          "refId": "A"
+        }
+      ],
+      "timeFrom": null,
+      "timeShift": null,
+      "title": "Application Modules Loaded",
+      "type": "stat"
+    }
+  ],
+  "refresh": "5s",
+  "schemaVersion": 26,
+  "style": "dark",
+  "tags": ["PromEx", "Application", "pinchflat"],
+  "templating": {
+    "list": [
+      {
+        "allValue": null,
+        "datasource": "prometheus",
+        "definition": "label_values(pinchflat_prom_ex_prom_ex_status_info, job)",
+        "hide": 0,
+        "includeAll": false,
+        "label": "Prometheus Job",
+        "multi": false,
+        "name": "job",
+        "options": [],
+        "query": "label_values(pinchflat_prom_ex_prom_ex_status_info, job)",
+        "refresh": 2,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 6,
+        "tagValuesQuery": "",
+        "tags": [],
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      },
+      {
+        "allValue": null,
+        "datasource": "prometheus",
+        "definition": "label_values(pinchflat_prom_ex_prom_ex_status_info, instance)",
+        "hide": 0,
+        "includeAll": false,
+        "label": "Application Instance",
+        "multi": false,
+        "name": "instance",
+        "options": [],
+        "query": "label_values(pinchflat_prom_ex_prom_ex_status_info{job=\"$job\"}, instance)",
+        "refresh": 2,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 0,
+        "tagValuesQuery": "",
+        "tags": [],
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timepicker": {
+    "refresh_intervals": ["5s", "10s", "30s", "1m", "5m"]
+  },
+  "timezone": "",
+  "title": "Pinchflat - PromEx Application Dashboard",
+  "uid": "7DBBC471C5775585391E8F24D1E62319",
+  "version": 1
+}

priv/repo/migrations/20250110231704_add_extractor_sleep_interval_to_settings.exs

@@ -0,0 +1,9 @@
+defmodule Pinchflat.Repo.Migrations.AddExtractorSleepIntervalToSettings do
+  use Ecto.Migration
+
+  def change do
+    alter table(:settings) do
+      add :extractor_sleep_interval_seconds, :number, null: false, default: 0
+    end
+  end
+end

test/pinchflat/downloading/download_option_builder_test.exs

@@ -287,7 +287,19 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
       assert {:sponsorblock_remove, "sponsor,intro"} in res
     end
 
-    test "does not include :sponsorblock_remove option without categories", %{media_item: media_item} do
+    test "includes :sponsorblock_mark option when specified", %{media_item: media_item} do
+      media_item =
+        update_media_profile_attribute(media_item, %{
+          sponsorblock_behaviour: :mark,
+          sponsorblock_categories: ["sponsor", "intro"]
+        })
+
+      assert {:ok, res} = DownloadOptionBuilder.build(media_item)
+
+      assert {:sponsorblock_mark, "sponsor,intro"} in res
+    end
+
+    test "does not include any sponsorblock option without categories", %{media_item: media_item} do
       media_item =
         update_media_profile_attribute(media_item, %{
           sponsorblock_behaviour: :remove,
@@ -296,9 +308,10 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
 
       assert {:ok, res} = DownloadOptionBuilder.build(media_item)
 
-      refute {:sponsorblock_remove, ""} in res
+      refute Keyword.has_key?(res, :sponsorblock_remove)
-      refute {:sponsorblock_remove, []} in res
+      refute Keyword.has_key?(res, :sponsorblock_mark)
       refute :sponsorblock_remove in res
+      refute :sponsorblock_mark in res
     end
 
     test "does not include any sponsorblock options when disabled", %{media_item: media_item} do
@@ -307,9 +320,10 @@ defmodule Pinchflat.Downloading.DownloadOptionBuilderTest do
 
       assert {:ok, res} = DownloadOptionBuilder.build(media_item)
 
-      refute {:sponsorblock_remove, ""} in res
+      refute Keyword.has_key?(res, :sponsorblock_remove)
-      refute {:sponsorblock_remove, []} in res
+      refute Keyword.has_key?(res, :sponsorblock_mark)
       refute :sponsorblock_remove in res
+      refute :sponsorblock_mark in res
     end
   end
 

test/pinchflat/downloading/media_download_worker_test.exs

@@ -127,6 +127,19 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
       end)
     end
 
+    test "does not set the job to retryable if youtube thinks you're a bot", %{media_item: media_item} do
+      expect(YtDlpRunnerMock, :run, 2, fn
+        _url, :get_downloadable_status, _opts, _ot, _addl -> {:ok, "{}"}
+        _url, :download, _opts, _ot, _addl -> {:error, "Sign in to confirm you're not a bot", 1}
+      end)
+
+      Oban.Testing.with_testing_mode(:inline, fn ->
+        {:ok, job} = Oban.insert(MediaDownloadWorker.new(%{id: media_item.id, quality_upgrade?: true}))
+
+        assert job.state == "completed"
+      end)
+    end
+
     test "it ensures error are returned in a 2-item tuple", %{media_item: media_item} do
       expect(YtDlpRunnerMock, :run, 2, fn
         _url, :get_downloadable_status, _opts, _ot, _addl -> {:ok, "{}"}

test/pinchflat/media_test.exs

@@ -921,6 +921,14 @@ defmodule Pinchflat.MediaTest do
       media_item = media_item_fixture()
       assert %Ecto.Changeset{} = Media.change_media_item(media_item)
     end
+
+    test "validates the title doesn't start with 'youtube video #'" do
+      # This is to account for youtube restricting indexing. See issue #549 for more
+      media_item = media_item_fixture()
+
+      assert %Ecto.Changeset{valid?: false} = Media.change_media_item(media_item, %{title: "youtube video #123"})
+      assert %Ecto.Changeset{valid?: true} = Media.change_media_item(media_item, %{title: "any other title"})
+    end
   end
 
   describe "change_media_item/1 when testing upload_date_index and source is a channel" do

test/pinchflat/settings_test.exs

@@ -77,5 +77,13 @@ defmodule Pinchflat.SettingsTest do
 
       assert %Ecto.Changeset{} = Settings.change_setting(setting, %{onboarding: true})
     end
+
+    test "ensures the extractor sleep interval is positive" do
+      setting = Settings.record()
+
+      assert %Ecto.Changeset{valid?: true} = Settings.change_setting(setting, %{extractor_sleep_interval_seconds: 1})
+      assert %Ecto.Changeset{valid?: true} = Settings.change_setting(setting, %{extractor_sleep_interval_seconds: 0})
+      assert %Ecto.Changeset{valid?: false} = Settings.change_setting(setting, %{extractor_sleep_interval_seconds: -1})
+    end
   end
 end

test/pinchflat/slow_indexing/file_follower_server_test.exs

@@ -1,7 +1,7 @@
 defmodule Pinchflat.SlowIndexing.FileFollowerServerTest do
   use Pinchflat.DataCase
 
-  alias alias Pinchflat.Utils.FilesystemUtils
+  alias Pinchflat.Utils.FilesystemUtils
   alias Pinchflat.SlowIndexing.FileFollowerServer
 
   setup do

test/pinchflat/slow_indexing/media_collection_indexing_worker_test.exs

@@ -57,25 +57,51 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       :ok
     end
 
-    test "it indexes the source if it should be indexed" do
+    test "indexes the source if it should be indexed" do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
+        {:ok, ""}
+      end)
+
       source = source_fixture(index_frequency_minutes: 10)
 
       perform_job(MediaCollectionIndexingWorker, %{id: source.id})
     end
 
-    test "it indexes the source no matter what if the source has never been indexed before" do
+    test "indexes the source no matter what if the source has never been indexed before" do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
+        {:ok, ""}
+      end)
+
       source = source_fixture(index_frequency_minutes: 0, last_indexed_at: nil)
 
       perform_job(MediaCollectionIndexingWorker, %{id: source.id})
     end
 
-    test "it indexes the source no matter what if the 'force' arg is passed" do
+    test "indexes the source no matter what if the 'force' arg is passed" do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
+        {:ok, ""}
+      end)
+
       source = source_fixture(index_frequency_minutes: 0, last_indexed_at: DateTime.utc_now())
 
       perform_job(MediaCollectionIndexingWorker, %{id: source.id, force: true})
     end
 
-    test "it does not do any indexing if the source has been indexed and shouldn't be rescheduled" do
+    test "doesn't use a download archive if the index has been forced" do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, opts, _ot, _addl_opts ->
+        refute :break_on_existing in opts
+        refute Keyword.has_key?(opts, :download_archive)
+
+        {:ok, ""}
+      end)
+
+      source =
+        source_fixture(collection_type: :channel, index_frequency_minutes: 0, last_indexed_at: DateTime.utc_now())
+
+      perform_job(MediaCollectionIndexingWorker, %{id: source.id, force: true})
+    end
+
+    test "does not do any indexing if the source has been indexed and shouldn't be rescheduled" do
       expect(YtDlpRunnerMock, :run, 0, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
         {:ok, ""}
       end)
@@ -85,7 +111,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       perform_job(MediaCollectionIndexingWorker, %{id: source.id})
     end
 
-    test "it does not reschedule if the source shouldn't be indexed" do
+    test "does not reschedule if the source shouldn't be indexed" do
       stub(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts -> {:ok, ""} end)
 
       source = source_fixture(index_frequency_minutes: -1)
@@ -94,7 +120,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       refute_enqueued(worker: MediaCollectionIndexingWorker, args: %{"id" => source.id})
     end
 
-    test "it kicks off a download job for each pending media item" do
+    test "kicks off a download job for each pending media item" do
       expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
         {:ok, source_attributes_return_fixture()}
       end)
@@ -105,7 +131,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       assert length(all_enqueued(worker: MediaDownloadWorker)) == 3
     end
 
-    test "it starts a job for any pending media item even if it's from another run" do
+    test "starts a job for any pending media item even if it's from another run" do
       expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
         {:ok, source_attributes_return_fixture()}
       end)
@@ -117,7 +143,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       assert length(all_enqueued(worker: MediaDownloadWorker)) == 4
     end
 
-    test "it does not kick off a job for media items that could not be saved" do
+    test "does not kick off a job for media items that could not be saved" do
       expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
         {:ok, source_attributes_return_fixture()}
       end)
@@ -130,7 +156,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       assert length(all_enqueued(worker: MediaDownloadWorker))
     end
 
-    test "it reschedules the job based on the index frequency" do
+    test "reschedules the job based on the index frequency" do
       source = source_fixture(index_frequency_minutes: 10)
       perform_job(MediaCollectionIndexingWorker, %{id: source.id})
 
@@ -141,7 +167,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       )
     end
 
-    test "it creates a task for the rescheduled job" do
+    test "creates a task for the rescheduled job" do
       source = source_fixture(index_frequency_minutes: 10)
 
       task_count_fetcher = fn ->
@@ -153,7 +179,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       end)
     end
 
-    test "it creates a future task for fast indexing if appropriate" do
+    test "creates a future task for fast indexing if appropriate" do
       source = source_fixture(index_frequency_minutes: 10, fast_index: true)
       perform_job(MediaCollectionIndexingWorker, %{id: source.id})
 
@@ -164,7 +190,7 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       )
     end
 
-    test "it deletes existing fast indexing tasks if a new one is created" do
+    test "deletes existing fast indexing tasks if a new one is created" do
       source = source_fixture(index_frequency_minutes: 10, fast_index: true)
       {:ok, job} = Oban.insert(FastIndexingWorker.new(%{"id" => source.id}))
       task = task_fixture(source_id: source.id, job_id: job.id)
@@ -174,14 +200,14 @@ defmodule Pinchflat.SlowIndexing.MediaCollectionIndexingWorkerTest do
       assert_raise Ecto.NoResultsError, fn -> Repo.reload!(task) end
     end
 
-    test "it does not create a task for fast indexing otherwise" do
+    test "does not create a task for fast indexing otherwise" do
       source = source_fixture(index_frequency_minutes: 10, fast_index: false)
       perform_job(MediaCollectionIndexingWorker, %{id: source.id})
 
       refute_enqueued(worker: FastIndexingWorker)
     end
 
-    test "it creates the basic media_item records" do
+    test "creates the basic media_item records" do
       expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
         {:ok, source_attributes_return_fixture()}
       end)

test/pinchflat/slow_indexing/slow_indexing_helpers_test.exs

@@ -14,6 +14,10 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
   alias Pinchflat.SlowIndexing.SlowIndexingHelpers
   alias Pinchflat.SlowIndexing.MediaCollectionIndexingWorker
 
+  setup do
+    {:ok, %{source: source_fixture()}}
+  end
+
   describe "kickoff_indexing_task/3" do
     test "schedules a job" do
       source = source_fixture(index_frequency_minutes: 1)
@@ -53,6 +57,16 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
       assert_in_delta DateTime.diff(job.scheduled_at, DateTime.utc_now(), :second), 0, 1
     end
 
+    test "schedules a job immediately if the user is forcing an index" do
+      source = source_fixture(index_frequency_minutes: 30, last_indexed_at: now_minus(5, :minutes))
+
+      assert {:ok, _} = SlowIndexingHelpers.kickoff_indexing_task(source, %{force: true})
+
+      [job] = all_enqueued(worker: MediaCollectionIndexingWorker, args: %{"id" => source.id})
+
+      assert_in_delta DateTime.diff(job.scheduled_at, DateTime.utc_now(), :second), 0, 1
+    end
+
     test "creates and attaches a task" do
       source = source_fixture(index_frequency_minutes: 1)
 
@@ -123,12 +137,6 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
   end
 
   describe "delete_indexing_tasks/2" do
-    setup do
-      source = source_fixture()
-
-      {:ok, %{source: source}}
-    end
-
     test "deletes slow indexing tasks for the source", %{source: source} do
       {:ok, job} = Oban.insert(MediaCollectionIndexingWorker.new(%{"id" => source.id}))
       _task = task_fixture(source_id: source.id, job_id: job.id)
@@ -172,13 +180,13 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
     end
   end
 
-  describe "index_and_enqueue_download_for_media_items/1" do
+  describe "index_and_enqueue_download_for_media_items/2" do
     setup do
       stub(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
         {:ok, source_attributes_return_fixture()}
       end)
 
-      {:ok, [source: source_fixture()]}
+      :ok
     end
 
     test "creates a media_item record for each media ID returned", %{source: source} do
@@ -282,6 +290,29 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
       assert %Ecto.Changeset{} = changeset
     end
 
+    test "doesn't blow up if the media item cannot be saved", %{source: source} do
+      stub(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, _opts, _ot, _addl_opts ->
+        response =
+          Phoenix.json_library().encode!(%{
+            id: "video1",
+            # This is a disallowed title - see MediaItem changeset or issue #549
+            title: "youtube video #123",
+            original_url: "https://example.com/video1",
+            live_status: "not_live",
+            description: "desc1",
+            aspect_ratio: 1.67,
+            duration: 12.34,
+            upload_date: "20210101"
+          })
+
+        {:ok, response}
+      end)
+
+      assert [changeset] = SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
+
+      assert %Ecto.Changeset{} = changeset
+    end
+
     test "passes the source's download options to the yt-dlp runner", %{source: source} do
       expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, opts, _ot, _addl_opts ->
         assert {:output, "/tmp/test/media/%(title)S.%(ext)S"} in opts
@@ -315,11 +346,7 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
     end
   end
 
-  describe "index_and_enqueue_download_for_media_items/1 when testing file watcher" do
+  describe "index_and_enqueue_download_for_media_items/2 when testing file watcher" do
-    setup do
-      {:ok, [source: source_fixture()]}
-    end
-
     test "creates a new media item for everything already in the file", %{source: source} do
       watcher_poll_interval = Application.get_env(:pinchflat, :file_watcher_poll_interval)
 
@@ -446,4 +473,62 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
       assert [] = SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
     end
   end
+
+  describe "index_and_enqueue_download_for_media_items when testing the download archive" do
+    test "a download archive is used if the source is a channel", %{source: source} do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, opts, _ot, _addl_opts ->
+        assert :break_on_existing in opts
+        assert Keyword.has_key?(opts, :download_archive)
+
+        {:ok, source_attributes_return_fixture()}
+      end)
+
+      SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
+    end
+
+    test "a download archive is not used if the source is not a channel" do
+      source = source_fixture(%{collection_type: :playlist})
+
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, opts, _ot, _addl_opts ->
+        refute :break_on_existing in opts
+        refute Keyword.has_key?(opts, :download_archive)
+
+        {:ok, source_attributes_return_fixture()}
+      end)
+
+      SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
+    end
+
+    test "a download archive is not used if the index has been forced to run" do
+      source = source_fixture(%{collection_type: :channel})
+
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, opts, _ot, _addl_opts ->
+        refute :break_on_existing in opts
+        refute Keyword.has_key?(opts, :download_archive)
+
+        {:ok, source_attributes_return_fixture()}
+      end)
+
+      SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source, was_forced: true)
+    end
+
+    test "the download archive is formatted correctly and contains the right video", %{source: source} do
+      media_items =
+        1..21
+        |> Enum.map(fn n ->
+          media_item_fixture(%{source_id: source.id, uploaded_at: now_minus(n, :days)})
+        end)
+
+      expect(YtDlpRunnerMock, :run, fn _url, :get_media_attributes_for_collection, opts, _ot, _addl_opts ->
+        archive_file = Keyword.get(opts, :download_archive)
+        last_media_item = List.last(media_items)
+
+        assert File.read!(archive_file) == "youtube #{last_media_item.media_id}"
+
+        {:ok, source_attributes_return_fixture()}
+      end)
+
+      SlowIndexingHelpers.index_and_enqueue_download_for_media_items(source)
+    end
+  end
 end

test/pinchflat/sources_test.exs

@@ -354,7 +354,56 @@ defmodule Pinchflat.SourcesTest do
     end
   end
 
-  describe "create_source/2 when testing options" do
+  describe "create_source/2 when testing yt-dlp options" do
+    test "sets use_cookies to true if the source has been set to use cookies" do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_source_details, _opts, _ot, addl ->
+        assert Keyword.get(addl, :use_cookies)
+
+        {:ok, playlist_return()}
+      end)
+
+      valid_attrs = %{
+        media_profile_id: media_profile_fixture().id,
+        original_url: "https://www.youtube.com/channel/abc123",
+        use_cookies: true
+      }
+
+      assert {:ok, %Source{}} = Sources.create_source(valid_attrs)
+    end
+
+    test "does not set use_cookies to false if the source has not been set to use cookies" do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_source_details, _opts, _ot, addl ->
+        refute Keyword.get(addl, :use_cookies)
+
+        {:ok, playlist_return()}
+      end)
+
+      valid_attrs = %{
+        media_profile_id: media_profile_fixture().id,
+        original_url: "https://www.youtube.com/channel/abc123",
+        use_cookies: false
+      }
+
+      assert {:ok, %Source{}} = Sources.create_source(valid_attrs)
+    end
+
+    test "skips sleep interval" do
+      expect(YtDlpRunnerMock, :run, fn _url, :get_source_details, _opts, _ot, addl ->
+        assert Keyword.get(addl, :skip_sleep_interval)
+
+        {:ok, playlist_return()}
+      end)
+
+      valid_attrs = %{
+        media_profile_id: media_profile_fixture().id,
+        original_url: "https://www.youtube.com/channel/abc123"
+      }
+
+      assert {:ok, %Source{}} = Sources.create_source(valid_attrs)
+    end
+  end
+
+  describe "create_source/2 when testing its options" do
     test "run_post_commit_tasks: false won't enqueue post-commit tasks" do
       expect(YtDlpRunnerMock, :run, &channel_mock/5)
 
@@ -902,28 +951,30 @@ defmodule Pinchflat.SourcesTest do
   end
 
   defp playlist_mock(_url, :get_source_details, _opts, _ot, _addl) do
-    {
+    {:ok, playlist_return()}
-      :ok,
+  end
+
+  defp channel_mock(_url, :get_source_details, _opts, _ot, _addl) do
+    {:ok, channel_return()}
+  end
+
+  defp playlist_return do
     Phoenix.json_library().encode!(%{
       channel: nil,
       channel_id: nil,
       playlist_id: "some_playlist_id_#{:rand.uniform(1_000_000)}",
       playlist_title: "some playlist name"
     })
-    }
   end
 
-  defp channel_mock(_url, :get_source_details, _opts, _ot, _addl) do
+  defp channel_return do
     channel_id = "some_channel_id_#{:rand.uniform(1_000_000)}"
 
-    {
-      :ok,
     Phoenix.json_library().encode!(%{
       channel: "some channel name",
       channel_id: channel_id,
       playlist_id: channel_id,
       playlist_title: "some channel name - videos"
     })
-    }
   end
 end

test/pinchflat/utils/number_utils_test.exs

@@ -47,4 +47,21 @@ defmodule Pinchflat.Utils.NumberUtilsTest do
       assert NumberUtils.human_byte_size(nil) == {0, "B"}
     end
   end
+
+  describe "add_jitter/2" do
+    test "returns 0 when the number is less than or equal to 0" do
+      assert NumberUtils.add_jitter(0) == 0
+      assert NumberUtils.add_jitter(-1) == 0
+    end
+
+    test "returns the number with jitter added" do
+      assert NumberUtils.add_jitter(100) in 100..150
+    end
+
+    test "optionally takes a jitter percentage" do
+      assert NumberUtils.add_jitter(100, 0.1) in 90..110
+      assert NumberUtils.add_jitter(100, 0.5) in 50..150
+      assert NumberUtils.add_jitter(100, 1) in 0..200
+    end
+  end
 end

test/pinchflat/yt_dlp/command_runner_test.exs

@@ -1,6 +1,7 @@
 defmodule Pinchflat.YtDlp.CommandRunnerTest do
   use Pinchflat.DataCase
 
+  alias Pinchflat.Settings
   alias Pinchflat.Utils.FilesystemUtils
 
   alias Pinchflat.YtDlp.CommandRunner, as: Runner
@@ -17,6 +18,12 @@ defmodule Pinchflat.YtDlp.CommandRunnerTest do
       assert {:ok, _output} = Runner.run(@media_url, :foo, [], "")
     end
 
+    test "considers a 101 exit code as being successful" do
+      wrap_executable("/app/test/support/scripts/yt-dlp-mocks/101_exit_code.sh", fn ->
+        assert {:ok, _output} = Runner.run(@media_url, :foo, [], "")
+      end)
+    end
+
     test "includes the media url as the first argument" do
       assert {:ok, output} = Runner.run(@media_url, :foo, [:ignore_errors], "")
 
@@ -89,6 +96,36 @@ defmodule Pinchflat.YtDlp.CommandRunnerTest do
     end
   end
 
+  describe "run/4 when testing sleep interval options" do
+    test "includes sleep interval options by default" do
+      Settings.set(extractor_sleep_interval_seconds: 5)
+
+      assert {:ok, output} = Runner.run(@media_url, :foo, [], "")
+
+      assert String.contains?(output, "--sleep-interval")
+      assert String.contains?(output, "--sleep-requests")
+      assert String.contains?(output, "--sleep-subtitles")
+    end
+
+    test "doesn't include sleep interval options when skip_sleep_interval is true" do
+      assert {:ok, output} = Runner.run(@media_url, :foo, [], "", skip_sleep_interval: true)
+
+      refute String.contains?(output, "--sleep-interval")
+      refute String.contains?(output, "--sleep-requests")
+      refute String.contains?(output, "--sleep-subtitles")
+    end
+
+    test "doesn't include sleep interval options when extractor_sleep_interval_seconds is 0" do
+      Settings.set(extractor_sleep_interval_seconds: 0)
+
+      assert {:ok, output} = Runner.run(@media_url, :foo, [], "")
+
+      refute String.contains?(output, "--sleep-interval")
+      refute String.contains?(output, "--sleep-requests")
+      refute String.contains?(output, "--sleep-subtitles")
+    end
+  end
+
   describe "run/4 when testing global options" do
     test "creates windows-safe filenames" do
       assert {:ok, output} = Runner.run(@media_url, :foo, [], "")

test/support/scripts/yt-dlp-mocks/101_exit_code.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+exit 101