Bumped version

* Format 'Scheduled At' in task table according to user's TZ

* Changed video preview layout to 1 column on all displays

* Made links in descriptions clickable

* Updated Copy JSON button to properly escape newline characters

README.md

@@ -53,7 +53,7 @@ If it doesn't work for your use case, please make a feature request! You can als
 - Self-contained - just one Docker container with no external dependencies
 - Powerful naming system so content is stored where and how you want it
 - Easy-to-use web interface with presets to get you started right away
-- First-class support for media center apps like Plex, Jellyfin, and Kodi
+- First-class support for media center apps like Plex, Jellyfin, and Kodi ([docs](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions#how-do-i-get-media-into-plexjellyfinkodi))
 - Supports serving RSS feeds to your favourite podcast app ([docs](https://github.com/kieraneglin/pinchflat/wiki/Podcast-RSS-Feeds))
 - Automatically downloads new content from channels and playlists
   - Uses a novel approach to download new content more quickly than other apps
@@ -63,7 +63,7 @@ If it doesn't work for your use case, please make a feature request! You can als
 - Allows automatically redownloading new media after a set period
   - This can help improve the download quality of new content or improve SponsorBlock tags
 - Optionally automatically delete old content ([docs](https://github.com/kieraneglin/pinchflat/wiki/Automatically-Delete-Media))
-- Advanced options like setting cutoff dates and filtering by title
+- Advanced options like setting cutoff dates and filtering by title ([docs](https://github.com/kieraneglin/pinchflat/wiki/Frequently-Asked-Questions#i-only-want-certain-videos-from-a-source---how-can-i-only-download-those))
 - Reliable hands-off operation
 - Can pass cookies to YouTube to download your private playlists ([docs](https://github.com/kieraneglin/pinchflat/wiki/YouTube-Cookies))
 - Sponsorblock integration
@@ -131,9 +131,6 @@ You _must_ ensure the host directories you've mounted are writable by the user r
 > [!IMPORTANT]
 > It's not recommended to run the container as root. Doing so can create permission issues if other apps need to work with the downloaded media.
 
-> [!TIP]
-> If you need to run any command as root, you can run `su` from the container's shell as there is no password set for the root user.
-
 ### ADVANCED: Storing Pinchflat config directory on a network share
 
 As pointed out in [#137](https://github.com/kieraneglin/pinchflat/issues/137), SQLite doesn't like being run in WAL mode on network shares. If you're running Pinchflat on a network share, you can disable WAL mode by setting the `JOURNAL_MODE` environment variable to `delete`. This will make Pinchflat run in rollback journal mode which is less performant but should work on network shares.
@@ -155,7 +152,7 @@ If you change this setting and it works well for you, please leave a comment on
 | `ENABLE_IPV6`               | No        | `false`                   | Setting to _any_ non-blank value will enable IPv6                                                                               |
 | `JOURNAL_MODE`              | No        | `wal`                     | Set to `delete` if your config directory is stored on a network share (not recommended)                                         |
 | `TZ_DATA_DIR`               | No        | `/etc/elixir_tzdata_data` | The container path where the timezone database is stored                                                                        |
-| `BASE_ROUTE_PATH`           | No        | `/`                       | The base path for route generation. Useful when running behind certain reverse proxies                                          |
+| `BASE_ROUTE_PATH`           | No        | `/`                       | The base path for route generation. Useful when running behind certain reverse proxies, but prefix must be stripped.            |
 | `YT_DLP_WORKER_CONCURRENCY` | No        | `2`                       | The number of concurrent workers that use `yt-dlp` _per queue_. Set to 1 if you're getting IP limited, otherwise don't touch it |
 
 ## EFF donations
@@ -164,9 +161,9 @@ Prior to 2024-05-10, a portion of all donations were given to the [Electronic Fr
 
 The EFF defends your online liberties and [backed](https://github.com/github/dmca/blob/9a85e0f021f7967af80e186b890776a50443f06c/2020/11/2020-11-16-RIAA-reversal-effletter.pdf) `youtube-dl` when Google took them down.
 
-## Pre-release disclaimer
+## Stability disclaimer
 
-This is pre-release software and anything can break at any time. I make not guarantees about the stability of this software, forward-compatibility of updates, or integrity (both related to and independent of Pinchflat). Essentially, use at your own risk and expect there will be rough edges for now.
+This software is in active development and anything can break at any time. I make no guarantees about the stability of this software, forward-compatibility of updates, or integrity (both related to and independent of Pinchflat).
 
 ## License
 

config/runtime.exs

@@ -79,11 +79,12 @@ if config_env() == :prod do
     tmpfile_directory: Path.join([System.tmp_dir!(), "pinchflat", "data"]),
     dns_cluster_query: System.get_env("DNS_CLUSTER_QUERY"),
     expose_feed_endpoints: expose_feed_endpoints,
-    timezone: System.get_env("TIMEZONE") || System.get_env("TZ") || "UTC",
+    # This is configured in application.ex
+    timezone: "UTC",
     log_path: log_path,
     base_route_path: base_route_path
 
-  config :tzdata, :data_dir, System.get_env("TZ_DATA_DIR", "/etc/elixir_tzdata_data")
+  config :tzdata, :data_dir, System.get_env("TZ_DATA_DIR", "/config/extras/elixir_tz_data")
 
   config :pinchflat, Pinchflat.Repo,
     database: db_path,

docker/selfhosted.Dockerfile

@@ -127,17 +127,6 @@ EXPOSE ${PORT}
 # Only copy the final release from the build stage
 COPY --from=builder /app/_build/${MIX_ENV}/rel/pinchflat ./
 
-# NEVER do this if you're running in an environment where you don't trust the user
-# (ie: most environments). This is only acceptable in a self-hosted environment.
-# The user could just run the whole container as root and bypass this anyway so
-# it's not a huge deal.
-# This removes the root password to allow users to assume root if needed. This is
-# preferrable to running the whole container as root so that the files/directories
-# created by the app aren't owned by root and are therefore easier for other users
-# and processes to interact with. If you want to just run the whole container as
-# root, use --user 0:0 or something.
-RUN passwd -d root
-
 HEALTHCHECK --interval=120s --start-period=10s \
   CMD curl --fail http://localhost:${PORT}/healthcheck || exit 1
 

lib/pinchflat/application.ex

@@ -4,9 +4,12 @@ defmodule Pinchflat.Application do
   @moduledoc false
 
   use Application
+  require Logger
 
   @impl true
   def start(_type, _args) do
+    check_and_update_timezone()
+
     children = [
       PinchflatWeb.Telemetry,
       Pinchflat.Repo,
@@ -47,4 +50,20 @@ defmodule Pinchflat.Application do
     :ok = Oban.Telemetry.attach_default_logger()
     :telemetry.attach_many("job-telemetry-broadcast", events, &PinchflatWeb.Telemetry.job_state_change_broadcast/4, [])
   end
+
+  # This has to be here (rather than runtime.exs) since the `tzdata` application
+  # has to be started before we can check the timezone
+  defp check_and_update_timezone do
+    attempted_timezone = System.get_env("TIMEZONE") || System.get_env("TZ") || "UTC"
+
+    valid_timezone =
+      if Tzdata.zone_exists?(attempted_timezone) do
+        attempted_timezone
+      else
+        Logger.warning("Invalid timezone #{attempted_timezone}, defaulting to UTC")
+        "UTC"
+      end
+
+    Application.put_env(:pinchflat, :timezone, valid_timezone)
+  end
 end

lib/pinchflat/downloading/media_retention_worker.ex

@@ -49,6 +49,8 @@ defmodule Pinchflat.Downloading.MediaRetentionWorker do
     end)
   end
 
+  # NOTE: Since this is a date and not a datetime, we can't add logic to have to-the-minute
+  # comparison like we can with retention periods. We can only compare to the day.
   defp delete_media_items_from_before_cutoff do
     deletable_media =
       MediaQuery.new()

lib/pinchflat/media/media.ex

@@ -15,6 +15,9 @@ defmodule Pinchflat.Media do
 
   alias Pinchflat.Lifecycle.UserScripts.CommandRunner, as: UserScriptRunner
 
+  # Some fields should only be set on insert and not on update.
+  @fields_to_drop_on_update [:playlist_index]
+
   @doc """
   Returns the list of media_items.
 
@@ -131,8 +134,6 @@ defmodule Pinchflat.Media do
   """
   def create_media_item_from_backend_attrs(source, media_attrs_struct) do
     attrs = Map.merge(%{source_id: source.id}, Map.from_struct(media_attrs_struct))
-    # Some fields should only be set on insert and not on update.
-    fields_to_drop_on_update = [:playlist_index]
 
     %MediaItem{}
     |> MediaItem.changeset(attrs)
@@ -140,7 +141,7 @@ defmodule Pinchflat.Media do
       on_conflict: [
         set:
           attrs
-          |> Map.drop(fields_to_drop_on_update)
+          |> Map.drop(@fields_to_drop_on_update)
           |> Map.to_list()
       ],
       conflict_target: [:source_id, :media_id]
@@ -153,8 +154,10 @@ defmodule Pinchflat.Media do
   Returns {:ok, %MediaItem{}} | {:error, %Ecto.Changeset{}}
   """
   def update_media_item(%MediaItem{} = media_item, attrs) do
+    update_attrs = Map.drop(attrs, @fields_to_drop_on_update)
+
     media_item
-    |> MediaItem.changeset(attrs)
+    |> MediaItem.changeset(update_attrs)
     |> Repo.update()
   end
 

lib/pinchflat/media/media_query.ex

@@ -88,7 +88,7 @@ defmodule Pinchflat.Media.MediaQuery do
       [mi, source],
       fragment("""
         IFNULL(retention_period_days, 0) > 0 AND
-        DATETIME('now', '-' || retention_period_days || ' day') > media_downloaded_at
+        DATETIME(media_downloaded_at, '+' || retention_period_days || ' day') < DATETIME('now')
       """)
     )
   end
@@ -100,8 +100,8 @@ defmodule Pinchflat.Media.MediaQuery do
       # downloaded_at minus the redownload_delay_days is before the upload date
       fragment("""
         IFNULL(redownload_delay_days, 0) > 0 AND
-        DATETIME('now', '-' || redownload_delay_days || ' day') > uploaded_at AND
-        DATETIME(media_downloaded_at, '-' || redownload_delay_days || ' day') < uploaded_at
+        DATE('now', '-' || redownload_delay_days || ' day') > DATE(uploaded_at) AND
+        DATE(media_downloaded_at, '-' || redownload_delay_days || ' day') < DATE(uploaded_at)
       """)
     )
   end

lib/pinchflat/sources/sources.ex

@@ -186,7 +186,13 @@ defmodule Pinchflat.Sources do
       {:ok, source_details} ->
         add_source_details_by_collection_type(source, changeset, source_details)
 
-      {:error, runner_error, _status_code} ->
+      err ->
+        runner_error =
+          case err do
+            {:error, error_msg, _status_code} -> error_msg
+            {:error, error_msg} -> error_msg
+          end
+
         Ecto.Changeset.add_error(
           changeset,
           :original_url,

lib/pinchflat/yt_dlp/media.ex

@@ -87,9 +87,11 @@ defmodule Pinchflat.YtDlp.Media do
   Returns the output template for yt-dlp's indexing command.
 
   NOTE: playlist_index is really only useful for playlists that will never change their order.
+  NOTE: I've switched back to `original_url` (from `webpage_url`) since it's started indicating
+        if something is a short via the URL again
   """
   def indexing_output_template do
-    "%(.{id,title,live_status,webpage_url,description,aspect_ratio,duration,upload_date,timestamp,playlist_index})j"
+    "%(.{id,title,live_status,original_url,description,aspect_ratio,duration,upload_date,timestamp,playlist_index})j"
   end
 
   @doc """
@@ -103,17 +105,17 @@ defmodule Pinchflat.YtDlp.Media do
       media_id: response["id"],
       title: response["title"],
       description: response["description"],
-      original_url: response["webpage_url"],
+      original_url: response["original_url"],
       livestream: !!response["live_status"] && response["live_status"] != "not_live",
       duration_seconds: response["duration"] && round(response["duration"]),
-      short_form_content: response["webpage_url"] && short_form_content?(response),
+      short_form_content: response["original_url"] && short_form_content?(response),
       uploaded_at: response["upload_date"] && parse_uploaded_at(response),
       playlist_index: response["playlist_index"] || 0
     }
   end
 
   defp short_form_content?(response) do
-    if String.contains?(response["webpage_url"], "/shorts/") do
+    if String.contains?(response["original_url"], "/shorts/") do
       true
     else
       # Sometimes shorts are returned without /shorts/ in the URL,

lib/pinchflat/yt_dlp/media_collection.ex

@@ -84,6 +84,7 @@ defmodule Pinchflat.YtDlp.MediaCollection do
          {:ok, parsed_json} <- Phoenix.json_library().decode(output) do
       {:ok, format_source_details(parsed_json)}
     else
+      {:error, %Jason.DecodeError{}} -> {:error, "Error decoding JSON response"}
       err -> err
     end
   end

lib/pinchflat_web/components/custom_components/text_components.ex

@@ -62,17 +62,24 @@ defmodule PinchflatWeb.CustomComponents.TextComponents do
   end
 
   @doc """
-  Renders a block of text with each line broken into a separate span.
+  Renders a block of text with each line broken into a separate span and links highlighted.
   """
   attr :text, :string, required: true
 
-  def break_on_newline(assigns) do
-    broken_text =
-      assigns.text
-      |> String.split("\n", trim: false)
-      |> Enum.intersperse(Phoenix.HTML.Tag.tag(:span, class: "inline-block mt-2"))
+  def render_description(assigns) do
+    formatted_text =
+      Regex.split(~r{https?://\S+}, assigns.text, include_captures: true)
+      |> Enum.map(fn
+        "http" <> _ = url ->
+          Phoenix.HTML.Tag.content_tag(:a, url, class: "text-blue-500 hover:text-blue-300", href: url, target: "_blank")
 
-    assigns = Map.put(assigns, :text, broken_text)
+        text ->
+          text
+          |> String.split("\n", trim: false)
+          |> Enum.intersperse(Phoenix.HTML.Tag.tag(:span, class: "inline-block mt-2"))
+      end)
+
+    assigns = Map.put(assigns, :text, formatted_text)
 
     ~H"""
     <span><%= @text %></span>

lib/pinchflat_web/components/layouts.ex

@@ -64,7 +64,7 @@ defmodule PinchflatWeb.Layouts do
           <.icon name={@icon} /> <%= @text %>
         </span>
         <span class="text-bodydark2">
-          <.icon name="hero-chevron-up" x-bind:class="{ 'rotate-180': selected }" />
+          <.icon name="hero-chevron-down" x-bind:class="{ 'rotate-180': selected }" />
         </span>
       </span>
 

lib/pinchflat_web/controllers/media_items/media_item_html/actions_dropdown.html.heex

@@ -2,7 +2,7 @@
   <:option>
     <span x-data="{ copied: false }" x-on:click={~s"
         copyWithCallbacks(
-          `#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_item))}`, 
+          String.raw`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_item))}`,
           () => copied = true, 
           () => copied = false
         )

lib/pinchflat_web/controllers/media_items/media_item_html/media_preview.heex

@@ -1,5 +1,5 @@
 <%= if media_type(@media_item) == :video do %>
-  <video controls class="max-h-96 w-full">
+  <video controls class="max-h-128 w-full">
     <source src={~p"/media/#{@media_item.uuid}/stream"} type="video/mp4" />
     Your browser does not support the video element.
   </video>

lib/pinchflat_web/controllers/media_items/media_item_html/show.html.heex

@@ -26,11 +26,12 @@
       <:tab title="Media" id="media">
         <div class="flex flex-col gap-10 dark:text-white">
           <%= if media_file_exists?(@media_item) do %>
-            <section class="grid grid-cols-1 xl:grid-cols-2 xl:gap-6 mt-6">
+            <section class="grid grid-cols-1 xl:gap-6 mt-6">
               <div>
                 <.media_preview media_item={@media_item} />
               </div>
-              <aside class="mt-4 xl:mt-0">
+              <aside class="mt-4">
+                <h2 class="text-xl mb-2"><%= @media_item.title %></h2>
                 <div>Uploaded: <%= DateTime.to_date(@media_item.uploaded_at) %></div>
                 <div>
                   <span :if={URI.parse(@media_item.original_url).scheme =~ "http"}>
@@ -44,7 +45,7 @@
                   </span>
                 </div>
                 <div :if={@media_item.description} class="mt-4 text-bodydark">
-                  <.break_on_newline text={@media_item.description} />
+                  <.render_description text={@media_item.description} />
                 </div>
               </aside>
             </section>
@@ -70,7 +71,7 @@
               <%= task.job.state %>
             </:col>
             <:col :let={task} label="Scheduled At">
-              <%= Calendar.strftime(task.job.scheduled_at, "%y-%m-%d %I:%M:%S %p %Z") %>
+              <.datetime_in_zone datetime={task.job.scheduled_at} />
             </:col>
           </.table>
         <% else %>

lib/pinchflat_web/controllers/media_profiles/media_profile_html/actions_dropdown.html.heex

@@ -2,7 +2,7 @@
   <:option>
     <span x-data="{ copied: false }" x-on:click={~s"
         copyWithCallbacks(
-          `#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_profile))}`, 
+          String.raw`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@media_profile))}`,
           () => copied = true, 
           () => copied = false
         )

lib/pinchflat_web/controllers/sources/source_html/actions_dropdown.html.heex

@@ -17,7 +17,7 @@
   <:option>
     <span x-data="{ copied: false }" x-on:click={~s"
         copyWithCallbacks(
-          `#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@source))}`, 
+          String.raw`#{Jason.Formatter.pretty_print(Phoenix.json_library().encode!(@source))}`,
           () => copied = true, 
           () => copied = false
         )

lib/pinchflat_web/controllers/sources/source_html/show.html.heex

@@ -67,7 +67,7 @@
               <%= task.job.state %>
             </:col>
             <:col :let={task} label="Scheduled At">
-              <%= Calendar.strftime(task.job.scheduled_at, "%y-%m-%d %I:%M:%S %p %Z") %>
+              <.datetime_in_zone datetime={task.job.scheduled_at} />
             </:col>
           </.table>
         <% else %>

mix.exs

@@ -4,7 +4,7 @@ defmodule Pinchflat.MixProject do
   def project do
     [
       app: :pinchflat,
-      version: "2024.10.2",
+      version: "2024.10.30",
       elixir: "~> 1.17",
       elixirc_paths: elixirc_paths(Mix.env()),
       start_permanent: Mix.env() == :prod,

test/pinchflat/downloading/media_retention_worker_test.exs

@@ -46,6 +46,23 @@ defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
       refute Repo.reload!(old_media_item).media_filepath
     end
 
+    test "deletes media files that are on their retention date per the 24-h clock" do
+      {_source, old_media_item, new_media_item} = prepare_records_for_retention_date(2)
+
+      just_over_two_days_ago = now_minus(2, :days) |> DateTime.add(-1, :minute)
+      just_under_two_days_ago = now_minus(2, :days) |> DateTime.add(1, :minute)
+
+      Media.update_media_item(old_media_item, %{media_downloaded_at: just_over_two_days_ago})
+      Media.update_media_item(new_media_item, %{media_downloaded_at: just_under_two_days_ago})
+
+      perform_job(MediaRetentionWorker, %{})
+
+      assert File.exists?(new_media_item.media_filepath)
+      refute File.exists?(old_media_item.media_filepath)
+      assert Repo.reload!(new_media_item).media_filepath
+      refute Repo.reload!(old_media_item).media_filepath
+    end
+
     test "sets culled_at and prevent_download" do
       {_source, old_media_item, new_media_item} = prepare_records_for_retention_date()
 
@@ -106,6 +123,25 @@ defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
       refute Repo.reload!(old_media_item).media_filepath
     end
 
+    # NOTE: Since this is a date and not a datetime, we can't add logic to have to-the-minute
+    # comparison like we can with retention periods. We can only compare to the day.
+    test "doesn't cull media from on or after the cutoff date" do
+      {_source, old_media_item, new_media_item} = prepare_records_for_source_cutoff_date(2)
+
+      Media.update_media_item(old_media_item, %{uploaded_at: now_minus(2, :days)})
+      Media.update_media_item(new_media_item, %{uploaded_at: now_minus(1, :day)})
+
+      perform_job(MediaRetentionWorker, %{})
+
+      assert File.exists?(new_media_item.media_filepath)
+      assert File.exists?(old_media_item.media_filepath)
+      assert Repo.reload!(new_media_item).media_filepath
+      assert Repo.reload!(old_media_item).media_filepath
+
+      refute Repo.reload!(new_media_item).culled_at
+      refute Repo.reload!(old_media_item).culled_at
+    end
+
     test "sets culled_at but not prevent_download" do
       {_source, old_media_item, new_media_item} = prepare_records_for_source_cutoff_date()
 
@@ -131,23 +167,6 @@ defmodule Pinchflat.Downloading.MediaRetentionWorkerTest do
       refute Repo.reload!(old_media_item).culled_at
     end
 
-    test "doesn't cull media from on or after the cutoff date" do
-      {_source, old_media_item, new_media_item} = prepare_records_for_source_cutoff_date(2)
-
-      Media.update_media_item(old_media_item, %{uploaded_at: now_minus(2, :days)})
-      Media.update_media_item(new_media_item, %{uploaded_at: now_minus(1, :day)})
-
-      perform_job(MediaRetentionWorker, %{})
-
-      assert File.exists?(new_media_item.media_filepath)
-      assert File.exists?(old_media_item.media_filepath)
-      assert Repo.reload!(new_media_item).media_filepath
-      assert Repo.reload!(old_media_item).media_filepath
-
-      refute Repo.reload!(new_media_item).culled_at
-      refute Repo.reload!(old_media_item).culled_at
-    end
-
     test "doesn't cull media items that have prevent_culling set" do
       {_source, old_media_item, _new_media_item} = prepare_records_for_source_cutoff_date()
 

test/pinchflat/fast_indexing/fast_indexing_helpers_test.exs

@@ -100,7 +100,7 @@ defmodule Pinchflat.FastIndexing.FastIndexingHelpersTest do
           Phoenix.json_library().encode!(%{
             id: "video2",
             title: "Video 2",
-            webpage_url: "https://example.com/shorts/video2",
+            original_url: "https://example.com/shorts/video2",
             live_status: "is_live",
             description: "desc2",
             aspect_ratio: 1.67,

test/pinchflat/media_test.exs

@@ -441,6 +441,13 @@ defmodule Pinchflat.MediaTest do
       assert Media.pending_download?(media_item)
     end
 
+    test "returns true if the cutoff date is equal to the upload date" do
+      source = source_fixture(%{download_cutoff_date: now_minus(2, :days)})
+      media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, uploaded_at: now_minus(2, :days)})
+
+      assert Media.pending_download?(media_item)
+    end
+
     test "returns false if there is a cutoff date after the media's upload date" do
       source = source_fixture(%{download_cutoff_date: now_minus(1, :day)})
       media_item = media_item_fixture(%{source_id: source.id, media_filepath: nil, uploaded_at: now_minus(2, :days)})
@@ -705,6 +712,21 @@ defmodule Pinchflat.MediaTest do
       assert media_item.media_filepath == update_attrs.media_filepath
     end
 
+    test "updating strips playlist_index from the provided attrs" do
+      media_item = media_item_fixture(playlist_index: 5)
+
+      update_attrs = %{
+        media_id: Faker.String.base64(12),
+        title: Faker.Commerce.product_name(),
+        media_filepath: "/video/#{Faker.File.file_name(:video)}",
+        source_id: source_fixture().id,
+        playlist_index: 1
+      }
+
+      assert {:ok, %MediaItem{} = media_item} = Media.update_media_item(media_item, update_attrs)
+      assert media_item.playlist_index == 5
+    end
+
     test "updating with invalid data returns error changeset" do
       media_item = media_item_fixture()
       assert {:error, %Ecto.Changeset{}} = Media.update_media_item(media_item, @invalid_attrs)

test/pinchflat/slow_indexing/slow_indexing_helpers_test.exs

@@ -188,7 +188,7 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
             description: "desc3",
             # Only focusing on these because these are passed to functions that
             # could fail if they're not present
-            webpage_url: nil,
+            original_url: nil,
             aspect_ratio: nil,
             duration: nil,
             upload_date: nil
@@ -299,7 +299,7 @@ defmodule Pinchflat.SlowIndexing.SlowIndexingHelpersTest do
           Phoenix.json_library().encode!(%{
             id: "video2",
             title: "Video 2",
-            webpage_url: "https://example.com/shorts/video2",
+            original_url: "https://example.com/shorts/video2",
             live_status: "is_live",
             description: "desc2",
             aspect_ratio: 1.67,

test/pinchflat/sources_test.exs

@@ -147,6 +147,18 @@ defmodule Pinchflat.SourcesTest do
       assert "could not fetch source details from URL" in errors_on(changeset).original_url
     end
 
+    test "adds an error if the runner succeeds but the result was invalid JSON" do
+      expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl -> {:ok, "Not JSON"} end)
+
+      valid_attrs = %{
+        media_profile_id: media_profile_fixture().id,
+        original_url: "https://www.youtube.com/channel/abc123"
+      }
+
+      assert {:error, %Ecto.Changeset{} = changeset} = Sources.create_source(valid_attrs)
+      assert "could not fetch source details from URL" in errors_on(changeset).original_url
+    end
+
     test "you can specify a custom custom_name" do
       expect(YtDlpRunnerMock, :run, &channel_mock/4)
 

test/pinchflat/yt_dlp/media_collection_test.exs

@@ -133,7 +133,7 @@ defmodule Pinchflat.YtDlp.MediaCollectionTest do
     test "returns an error if the output is not JSON" do
       expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot, _addl_opts -> {:ok, "Not JSON"} end)
 
-      assert {:error, %Jason.DecodeError{}} = MediaCollection.get_source_details(@channel_url)
+      assert {:error, "Error decoding JSON response"} = MediaCollection.get_source_details(@channel_url)
     end
   end
 

test/pinchflat/yt_dlp/media_test.exs

@@ -139,7 +139,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
   describe "indexing_output_template/0" do
     test "contains all the greatest hits" do
       attrs =
-        ~w(id title live_status webpage_url description aspect_ratio duration upload_date timestamp playlist_index)a
+        ~w(id title live_status original_url description aspect_ratio duration upload_date timestamp playlist_index)a
 
       formatted_attrs = "%(.{#{Enum.join(attrs, ",")}})j"
 
@@ -153,7 +153,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
         "id" => "TiZPUDkDYbk",
         "title" => "Trying to Wheelie Without the Rear Brake",
         "description" => "I'm not sure what I expected.",
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "live_status" => "not_live",
         "aspect_ratio" => 1.0,
         "duration" => 60,
@@ -177,7 +177,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "sets short_form_content to true if the URL contains /shorts/" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/shorts/TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/shorts/TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 61,
         "upload_date" => "20210101"
@@ -188,7 +188,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "sets short_form_content to true if the aspect ratio are duration are right" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 0.5,
         "duration" => 59,
         "upload_date" => "20210101"
@@ -199,7 +199,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "sets short_form_content to false otherwise" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 61,
         "upload_date" => "20210101"
@@ -210,7 +210,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "doesn't blow up if short form content-related fields are missing" do
       response = %{
-        "webpage_url" => nil,
+        "original_url" => nil,
         "aspect_ratio" => nil,
         "duration" => nil,
         "upload_date" => "20210101"
@@ -221,7 +221,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "parses the duration" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 60.4,
         "upload_date" => "20210101"
@@ -232,7 +232,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "doesn't blow up if duration is missing" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => nil,
         "upload_date" => "20210101"
@@ -243,7 +243,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "sets livestream to false if the live_status field isn't present" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 60,
         "upload_date" => "20210101"
@@ -254,7 +254,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "doesn't blow up if playlist_index is missing" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => nil,
         "upload_date" => "20210101"
@@ -267,7 +267,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
   describe "response_to_struct/1 when testing uploaded_at" do
     test "parses the upload date from the timestamp if present" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 61,
         "upload_date" => "20210101",
@@ -281,7 +281,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "parses the upload date from the uploaded_at if timestamp is present but nil" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 61,
         "upload_date" => "20210101",
@@ -295,7 +295,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "parses the upload date from the uploaded_at if timestamp absent" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 61,
         "upload_date" => "20210101"
@@ -308,7 +308,7 @@ defmodule Pinchflat.YtDlp.MediaTest do
 
     test "doesn't blow up if upload date is missing" do
       response = %{
-        "webpage_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
+        "original_url" => "https://www.youtube.com/watch?v=TiZPUDkDYbk",
         "aspect_ratio" => 1.0,
         "duration" => 61,
         "upload_date" => nil

test/support/fixtures/media_fixtures.ex

@@ -95,7 +95,7 @@ defmodule Pinchflat.MediaFixtures do
     media_attributes = %{
       id: "video1",
       title: "Video 1",
-      webpage_url: "https://example.com/video1",
+      original_url: "https://example.com/video1",
       live_status: "not_live",
       description: "desc1",
       aspect_ratio: 1.67,

test/support/fixtures/sources_fixtures.ex

@@ -80,7 +80,7 @@ defmodule Pinchflat.SourcesFixtures do
       %{
         id: "video1",
         title: "Video 1",
-        webpage_url: "https://example.com/video1",
+        original_url: "https://example.com/video1",
         live_status: "not_live",
         description: "desc1",
         aspect_ratio: 1.67,
@@ -90,7 +90,7 @@ defmodule Pinchflat.SourcesFixtures do
       %{
         id: "video2",
         title: "Video 2",
-        webpage_url: "https://example.com/video2",
+        original_url: "https://example.com/video2",
         live_status: "is_live",
         description: "desc2",
         aspect_ratio: 1.67,
@@ -100,7 +100,7 @@ defmodule Pinchflat.SourcesFixtures do
       %{
         id: "video3",
         title: "Video 3",
-        webpage_url: "https://example.com/video3",
+        original_url: "https://example.com/video3",
         live_status: "not_live",
         description: "desc3",
         aspect_ratio: 1.0,