Compare commits

..

126 Commits

Author SHA1 Message Date
Kieran Eglin 21eaeb8b60 Bumped version 2024-03-28 17:02:38 -07:00
Kieran Eglin 2b348607bf Combined old and new GH actions for docker 2024-03-28 16:40:50 -07:00
Kieran Eglin 2affcd8616 Testing conditionals 2024-03-28 16:32:54 -07:00
Kieran cab4c339e5 [Housekeeping] Add release workflow (takeover) (#147)
* [Housekeeping] Add release workflow (#141)

* add release workflow

This will create tags when a new release is published

* fix(workflows): refactor release workflow

Signed-off-by: Devin Buhl <devin@buhl.casa>

* Update release.yml

---------

Signed-off-by: Devin Buhl <devin@buhl.casa>

* Linting

* Testing changes

---------

Signed-off-by: Devin Buhl <devin@buhl.casa>
Co-authored-by: Devin Buhl <onedr0p@users.noreply.github.com>
2024-03-28 16:14:54 -07:00
Kieran 103541cd20 [Enhancement] Start tracking a media item's duration (#146)
* Added duration field, started importing it during indexing and download

* Added duration to RSS feed
2024-03-28 15:51:04 -07:00
Kieran Eglin 8cba501bff fixed typo 2024-03-28 15:11:12 -07:00
Kieran 6a02f217e3 [Triage] Allow changing SQLite journal mode (#145)
* Adds ability to configure journal mode for SQLite

* docs

* docs
2024-03-28 15:10:27 -07:00
Kieran c3f49184b5 [Feature] Add SponsorBlock-based section removal (#144)
* Added sponsorblock columns to media profile

* Added sponsorblock options to profile form

* Added SB to download options builder
2024-03-28 12:02:13 -07:00
Kieran 3837a0dedd trying something new with queries (#142) 2024-03-28 08:59:53 -07:00
Kieran Eglin c70f211232 Bumped version 2024-03-27 12:59:58 -07:00
Kieran ba0ded03a4 Updated fast indexing to respect title filters (#136) 2024-03-27 12:39:50 -07:00
Kieran Eglin 8513deef09 Hopefully fixed select styling 2024-03-27 12:28:25 -07:00
Kieran Eglin 83b9adea59 Turned GH actions inputs into dropdowns 2024-03-27 12:10:53 -07:00
Kieran Eglin 10a639a976 Fixed copy not working in insecure contexts 2024-03-27 12:07:15 -07:00
Kieran c79e27d9b3 Disabled index frequency when fast indexing is enabled (#135) 2024-03-27 11:36:35 -07:00
Kieran 8ef5e0f41f Added total downloaded count to source view (#134) 2024-03-27 10:27:48 -07:00
Kieran 854502634c RSS feed for sources (#110)
* Add media streaming (#108)

* [WIP] set up streaming route

* Added UUID to sources and media items

* Added media preview to MI show page

* Added plug to strip file extensions

* [VERY WIP] basic podcast RSS setup

* [WIP] got basic podcast RSS working

* [WIP] more expanding on RSS

* Comment

* [WIP] Working on refactoring feed

* Added UUID backfill to a migration

* [WIP] Moar refactoring

* [WIP] Adding UI for getting RSS feed

* Many tests

* Added conditional routing for feed URLs

* Removed the need for url_base to be set

* Updated preset name

* Rendered certain fields HTML-safe; Added logging to confirm range request support

* Fixed incorrect scheme issue

* Updated env var

* Updated other UI to use dropdown

* removed commented code

* Generate rss feeds (#123)

* Added plug to strip file extensions

* [VERY WIP] basic podcast RSS setup

* [WIP] got basic podcast RSS working

* [WIP] more expanding on RSS

* [WIP] Working on refactoring feed

* Added UUID backfill to a migration

* [WIP] Moar refactoring

* [WIP] Adding UI for getting RSS feed

* Many tests

* Added conditional routing for feed URLs

* Removed the need for url_base to be set

* Updated preset name

* Rendered certain fields HTML-safe; Added logging to confirm range request support

* Fixed incorrect scheme issue

* Updated env var

* Updated other UI to use dropdown

* removed commented code

* docs

* Added unique index to UUID fields
2024-03-27 10:11:25 -07:00
Kieran Eglin fc8eed8b50 Bumped version 2024-03-26 08:55:04 -07:00
Kieran f42aab286a [Bugfix] Expose port in Dockerfile and force IPv4 (#129)
* Exposes port on built image

* Forced IPv4

* Added current version to sidebar
2024-03-26 08:54:20 -07:00
Kieran bf11cd0dd9 Added description to source (#124) 2024-03-25 15:40:09 -07:00
Kieran Eglin e051a37f0d Pushed bug comment 2024-03-25 12:15:21 -07:00
Kieran Eglin 64ebf756ab Bumped version 2024-03-25 11:56:31 -07:00
Kieran 377bc617ea Removed unneeded startup tasks (#121) 2024-03-25 11:52:02 -07:00
Kieran a03f69a8d4 Started storing source images in internal metadata folder (#120) 2024-03-25 11:36:03 -07:00
Kieran d39ebe7ac1 Improve default recommendation for artist placeholder (#119)
* Improved default recommendation for audio artist placeholder

* Updated docs
2024-03-25 10:34:22 -07:00
Kieran 8b5dcc15d6 Fixed filepath bug when parsing metadata (#118) 2024-03-25 10:20:29 -07:00
Kieran 5e58dcd879 Cookie auth (#116)
* Added new extras directory

* Automatically create cookie file on boot

* Added cookie file option to DL option builder

* Put cookie auth in correct spot

* Updated Docs

* Removed useless setter for config path
2024-03-24 20:23:25 -07:00
Kieran Eglin a9d5407071 Bumped version 2024-03-24 15:42:03 -07:00
Kieran 37c1da8014 Added GHCR to README (#115) 2024-03-24 15:40:16 -07:00
Kieran Eglin 937a14985f One more GHCR shot 2024-03-24 15:15:43 -07:00
Kieran Eglin a6a20c111f Added GHCR tags 2024-03-24 15:13:12 -07:00
Kieran Eglin 2bb96edc00 Merge branch 'master' of https://github.com/kieraneglin/pinchflat 2024-03-24 15:07:13 -07:00
Kieran Eglin 87f07ff634 Testing GHCR support 2024-03-24 15:07:00 -07:00
Kieran dd60cb5419 Improve preflight file permissions check (#114)
* Added startup permissions check with more helpful message

* Updated README

* Fixes faulty test
2024-03-24 15:06:08 -07:00
Kieran Eglin 610b00566b Added logging for job reset task 2024-03-24 09:28:05 -07:00
Kieran aba15ed886 Adds a method for retrying stuck jobs (#112) 2024-03-24 09:21:01 -07:00
Kieran Eglin 87da716fdf Adds platform option when building docker containers 2024-03-24 09:20:07 -07:00
Kieran Eglin 45f056c6e0 Enabled perf profiler for erlang JIT 2024-03-22 23:34:56 -07:00
Kieran Eglin 11d160008c Updated Docker image generation for GH action 2024-03-22 23:15:51 -07:00
Kieran Eglin ba0a1ba100 Out of alpha, into beta 2024-03-22 12:43:38 -07:00
Kieran Eglin 8c51a286c9 Bumped version 2024-03-21 12:27:14 -07:00
Kieran 46c27e4875 Add docs link (#103)
* Added docs link to README

* Added docs to sidebar in-app

* Updated some tests

* Updated docs in media profile form
2024-03-21 12:26:51 -07:00
Kieran Eglin b639918555 Updated example worker path in README 2024-03-20 17:05:34 -07:00
Kieran Eglin 5596351f58 Renamed 'music' in README to 'audio content' 2024-03-20 16:28:14 -07:00
Kieran ad150b6f16 Improve README (#101)
* Started improving README

* Adds table of contents

* Updated PR template
2024-03-20 16:26:44 -07:00
Kieran Eglin f6d8e15670 Bumped version 2024-03-20 14:48:58 -07:00
Kieran 2906ff1b30 Update logo (#100)
* Placeholder commit

* Added new logo to app

* added logo to header of small screens

* Updated favicon

* Added logo to README

* Updated README
2024-03-20 14:34:49 -07:00
Kieran 0582a7bfd5 Misc indexing improvements (#99)
* increased file follower timeout to 10 minutes

* Removed unique index across a source's collection and its media profile

* Ensure source gets updating during slow indexing
2024-03-20 11:34:33 -07:00
Kieran 3db2e8190f Source title regex filtering (#98)
* Added a basic/advanced mode to source form

* Added regex field and UI to source form

* Implemented title regex filtering
2024-03-20 10:51:14 -07:00
Kieran Eglin 087c9c2d0f Bumped version; added link to donation receipts 2024-03-19 14:32:59 -07:00
Kieran c046abf36f Improve index tables and improved consistency (#97)
* Simplifies index pages

* Improved UX of more pages
2024-03-19 14:03:02 -07:00
Kieran 70dd95211f Download images for sources (#96)
* [WIP] hooked up photo downloading to source metadata worker

* Added new attributes for source images

* Added option to profile form
2024-03-19 13:34:01 -07:00
Kieran 8491f8b317 Source NFO downloads (#95)
* Hooked up series directory finding to source metadata runner

* Fixed aired NFO tag for episodes

* Updated MI NFO builder to take in a filepath

* Hooked up NFO generation to the source worker

* Added NFO controls to form

* Improved the way the source metadata worker updates the source

* Consolidated NFO selection options in media profile instead of source
2024-03-18 14:27:28 -07:00
Kieran Eglin d0cb782082 Bumped version 2024-03-15 13:35:20 -07:00
Kieran 7a7e8080cd Misc refactors 2024-03-15 (#90)
* Fixed TV show output template

* Discard jobs if record not found; hopefully addressed timing-related source deletion bug
2024-03-15 12:29:02 -07:00
Kieran fbe21cb304 Misc refactors 2024-03-14 (#89)
* Adds method to improve cleanup of empty directories

* resolved bug where source metadata worker could call itself in an infinite loop

* Refactored file deletion for media items

* Removed useless filesystem data worker

* Updated task listing fns to take a record directly

* Refactored the way I call workers

* Improved some tests
2024-03-15 10:44:58 -07:00
Kieran 0f3329e97d Improve episode-level compatability with media center apps (#86)
* Add media profile presets (#85)

* Added presets for output templates

* Added presets for the entire media profile form

* Append `-thumb` to thumbnails when downloading (#87)

* Appended -thumb to thumbnails when downloading

* Added code to compensate for yt-dlp bug

* Squash all the commits from the other branch bc I broke things (#88)
2024-03-14 12:30:08 -07:00
Kieran 25eb772896 Add support for episode NFO files (#84)
* Added nfo builder for 'episodes'

* Added NFO download fields; hooked up NFO generation to downloading pipeline

* Added NFO option to media profile
2024-03-13 11:31:53 -07:00
Kieran cf59bf99cd More onboarding improvements (#83)
* Improved custom YYYY-MM-DD output template option

* Clarified embedding vs. downloading on media profile form

* Improved form helpers more; Added a helper to every field
2024-03-13 08:51:40 -07:00
Kieran Eglin 47e96e3780 Bumped version 2024-03-12 16:25:01 -07:00
Kieran a7af6a9125 [Bugfix] Fix reddit bugs v2 (#82)
* Ensured thumbnail is converted to jpg before embedding

* Ensured media indexing doesn't fail if an upload date can't be parsed
2024-03-12 16:23:51 -07:00
Kieran 8f9d18dc71 Added input validation to help with cutoff date usage (#80) 2024-03-12 15:31:50 -07:00
Kieran Eglin 513212faf2 Bumped version 2024-03-12 12:19:06 -07:00
Kieran 5ec2c92a0c [Bugfix] Fixes issue with grabbing source/media details when first video is a premier (#79)
* Fixed issue with source details when first video is a premier

* Updated other occurance
2024-03-12 12:18:19 -07:00
Kieran 3c897e96e6 Refactor modules into contexts (#78)
* [WIP] break out a few contexts, start refactoring fast index modules

* [WIP] more contexts, this time around slow indexing and downloads

* [WIP] got all tests passing

* [WIP] Added moduledocs

* Built a genserver to rename old jobs on boot

* Added a module naming check; moved things around

* Fixed specs
2024-03-12 10:54:55 -07:00
Kieran Eglin c0885cfaf2 Bumped version 2024-03-11 15:42:18 -07:00
Kieran b2f39a7b7f Onboarding improvements (#76)
* Disabled pro modal during onboarding

* Restructured the way onboarding changes settings

* Added better upload date template placeholder
2024-03-11 15:19:10 -07:00
Kieran bada14625b Adds local details to local dockerfile (#75) 2024-03-11 15:05:19 -07:00
Kieran 7e4f1a8412 Improve audio-only download (#74)
* Improved quality from auto-only download; enables thumbnail and data embedding for audio

* Updated UI to reflect new audio behaviour
2024-03-11 14:33:27 -07:00
Kieran 0e537d8f57 Upgrade to Elixir 1.16.2 (#71)
* Upgraded local docker to 1.16.2

* Removed tmpfix from mix check that was waiting for 1.16.2

* Update selfhosted dockerfile with some addl. updates
2024-03-11 11:47:13 -07:00
Kieran afe41d2e36 Converts thumbnails to jpg for better compat (#70) 2024-03-11 09:03:40 -07:00
Kieran Eglin f36bd5abab Bumped version to the correct one LMAO 2024-03-10 21:35:31 -07:00
Kieran Eglin b3f094fa69 Bumped version 2024-03-10 21:33:59 -07:00
Kieran ccdcf8eec5 Download cutoff date for sources (#69)
* Added new uploaded_at column to media items

* Updated indexer to pull upload date

* Updates media item creation to update on conflict

* Added download cutoff date to sources

* Applies cutoff date logic to pending media logic

* Updated docs
2024-03-10 21:24:01 -07:00
Kieran 67d7f397d1 Updated check origin in release environment (#67) 2024-03-10 15:31:40 -07:00
Kieran 09a4bcb36b Local data worker improvements (#66)
* Adds new worker for backfilling data

* Adds backfill job to startup tasks
2024-03-10 15:17:15 -07:00
Kieran dc0313d875 Fast indexing (#58)
* Made method to getting singular media details; Renamed other related method

* Takes a fun and flirty digression to remove abstractions around yt-dlp since I'm 100% committed to using it exclusively

* Removed commented test code

* Lays the groundwork for fast indexing

* Added module for working with youtube RSS feed

* Added methods to kick off indexing workers from RSS response

* Improve short detection (#59)

* Made media attribute-related yt-dlp calls return a struct

* Added shorts attribute to media items

* Added ability to discern a short from yt-dlp response

* Updated search to use new shorts attribute

* Fast index UI (#63)

* Added fast_index field and adds it to source form

* Added fast indexing to source changeset operations

* Added fast indexing worker and updated other modules to start using it

* Handled fast index worker on source update

* Add support modals (#65)

* Added fast indexing upgrade modal

* Improved modal on smaller screens

* Updated links to work again

* Added donation modal

* Reverted source fast index to 15 minutes

* Removed unneeded HTML attributes from old alpine approach
2024-03-10 14:36:34 -07:00
Kieran 1caddd86c7 Resolved bug where null values in search terms would break search (#57) 2024-03-07 12:24:20 -08:00
Kieran 1c65b41002 [Bugfix] Runtime log level fix (#56)
* Added log level to runtime config

* Started comparing against an atom instead of a string
2024-03-07 11:26:20 -08:00
Kieran 6e195f6a6f Added log level to runtime config (#55) 2024-03-07 11:07:00 -08:00
Kieran d0f55cd463 Add settings model (#54)
* Adds a basic settings model

* Added more settings methods; Hooked up initial settings runner to app boot

* Update onboarding flow to use settings model instead of session data
2024-03-07 10:32:12 -08:00
Kieran 39caf4a94f Added image tagging input to GH actions (#53) 2024-03-04 13:37:15 -08:00
Kieran 254054c0c1 [Bugfix] Fix basic auth in deployed apps (#52)
* Renamed auth env vars and updated them to work in prod env

* Updated basic auth parsing to ignore empty strings
2024-03-04 13:29:26 -08:00
Kieran 5c9933b684 Updated README (#51) 2024-03-04 11:25:33 -08:00
Kieran cbade0dd02 Rename video-related modules to be media-related (#50)
* Added test from before that I forgor

* Renamed video-related modules to media
2024-03-04 10:46:33 -08:00
Kieran 63e5058365 Streaming media item creation during indexing (#49)
* Implemented streaming during indexing

* Updated file watcher to enqueue download; refactored download worker methods

* Updated File Follower Server timeout
2024-03-04 10:14:02 -08:00
Kieran a46dd19ec7 Added way to track media filesize (#48) 2024-03-02 19:19:10 -08:00
Kieran 55f9676bb7 Clarifies wording of source form for index frequency (#47) 2024-03-02 13:09:58 -08:00
Kieran fc52f16475 Support audio only downloads (#46)
* Adds option to only download audio

* Improves helper text for profile options
2024-03-02 13:01:47 -08:00
Kieran 8cc36928f1 Source attributes in output template (#45)
* Updated download options to take a media item; allowed for specifying custom output path template options

* Fixed bug where indexing job wouldn't run for the first time

* Renamed friendly_name to custom_name

* Added new options to default template and UI

* Improved explainer UI
2024-03-02 12:15:10 -08:00
Kieran 1ad9e2c331 Pre-prepare for alpha launch (#44)
* Add an initial version

* Add a license
2024-03-01 16:47:19 -08:00
Kieran 6738071613 Useful data on homepage v1 (#43)
* Made the homepage _slightly_ more useful

* Very minor refactor in alpine body-level object
2024-03-01 16:25:55 -08:00
Kieran b54e69c609 Improve relationship UI handling (#42)
* Added tabbed view; improved relationships for media profile

* Adds new maybe_limit method for queries

* Improves UI for media items, associated tasks
2024-02-29 15:35:51 -08:00
Kieran 5b29f5036a Responsive UI improvements (#41)
* Improved responsive UX for media profiles

* Improved responsiveness for other pages

* Added docs; fixed up stragglers
2024-02-29 12:24:54 -08:00
Kieran 9fc2edfd4a Added some logos up in this (#40) 2024-02-29 11:37:14 -08:00
Kieran 01b8856f22 UX improvements v1 (#39)
* Removed collection_type user input instead inferring from yt-dlp response

* Updated docs

* Added delete buttons for source; Refactored the way deletion methods work

* Update source to always run an initial index

* Added deletion to the last models

* Improved clarity around deletion operation

* Improved task fetching and deletion methods

* More tests
2024-02-28 09:54:01 -08:00
Kieran 3bb7edba0b Resolved windows filesystem bug (#38) 2024-02-28 08:56:25 -08:00
Kieran 52c0a1742a Improve metadata storage (#37)
* Updated config path to specify metadata storage path

* Removed metadata column from DB, replacing it with filepath columns

* Updated app to store compressed metadata; automatically download thumbnails

* Ensured metadata is deleted when other files are deleted

* Updated docs

* Added inets to application start so http calls work in prod
2024-02-24 13:40:05 -08:00
Kieran e33a46bddc Small form improvements (#36)
* Clarified intent of custom name field

* More copy improvements
2024-02-22 15:48:25 -08:00
Kieran 2ca395a5bf Updated video downloader to use full URL; improved docs (#35) 2024-02-22 12:46:08 -08:00
Kieran Eglin ebebbf7571 Updated docker build to only run on dispatch 2024-02-22 12:30:09 -08:00
Kieran b8cec51c3b Docker build action (round 2) (#34)
* Added first attempt at dockerfile

* Added workflow dispatch [skip ci]

* Trying PR trigger just for now [skip ci]

* Trying some more [skip ci]

* Revert for merge
2024-02-21 15:19:06 -08:00
Kieran 0d29f70a28 Added log rotation for prod config (#32) 2024-02-21 12:01:18 -08:00
Kieran c1f30dbd2c Added optional basic auth (#31) 2024-02-21 11:25:25 -08:00
Kieran fdb83ed3f3 Improve dev tooling (#30)
* added ex_check and got it passing

* Set up ex_check to run in test env

* Added static code analysis

* Updated actions file

* Updated dockerfile
2024-02-21 10:28:27 -08:00
Kieran 94f777d1ed Updated source's friendly_name to default to the collection_name (#29) 2024-02-20 14:24:47 -08:00
Kieran 47f0fd533f Refactor and document output path (#28)
* Updated output path builder to better use yt-dlp variables

* Added UI explaining output template options

* Fixed outdated reference in dockerfile
2024-02-20 13:40:13 -08:00
Kieran 7d62f7c7df Rename media sources (#25)
* Renamed MediaSource to Sources

* Renamed MediaItem controller namespace
2024-02-20 09:48:20 -08:00
Kieran d4d653398b Basic onboading flow (#24)
* removed unused file

* [WIP] Created onboarding form

* Finished basic onboarding flow; added tests

* Improved onboarding language
2024-02-20 09:37:17 -08:00
Kieran fab77f4deb Prod packaging v1 (#23)
* Sets media directory for prod

* Improved logging; Added docker files for prod

* Improved fetching SECRET_KEY_BASE for selfhosted instances
2024-02-16 11:50:00 -08:00
Kieran 8fcd41f45d Switch to sqlite (#22)
* Updated project to use sqlite

* Edited migrations directly because I fear no man

* Updated search functions and tests to work with sqlite

* Updated build tools to remove postgres
2024-02-16 10:11:41 -08:00
Kieran 44bd105e80 Add resolution option (#21)
* added preferred resolution to profiles

* Add resolution to download option builder
2024-02-15 14:21:41 -08:00
Kieran 9d3b9cc063 Delete media items (#20)
* Added method for deleting media files and their content

* Adds controllers and methods for deleting media and files

* Improved tmpfile setup and teardown for tests

* Actually got tmpfile cleanup running once per suite run

* Finally fixed flash messages
2024-02-15 13:51:19 -08:00
Kieran 060e340558 Search v1 (#19)
* Adds description to media items; hooks it up to indexing/media downloading

* Added a search method using postgres fulltext search

* Hooked up search functionality to the search form

* Added persistence to the search form when on search page
2024-02-13 14:46:14 -08:00
Kieran 219320ce11 Media item lifecycle improvements (#18)
* Added jobs for enqueuing/dequeuing media based on source's status

* adds lifecycle columns to source and media_item tables
2024-02-09 20:51:23 -08:00
Kieran 8bdd189fa5 Media UI v1 (#17)
* Updated comments

* Added basic media tables to sources#show

* Adds a very basic listing and show page for media
2024-02-09 16:21:51 -08:00
Kieran dabec62e99 Redo indexing mechanism (#16)
* Bumped up the line length because I fear no man

* Refactored indexing

Previously, indexing worked by collecting the video IDs of only videos
that matched indexing criteria. This new model instead stores ALL videos
for a given source, but will only _download_ videos that meet that criteria.
This lets us backfill without indexing, makes it easier to add in other
backends, lets us download one-off videos for a source that don't quite
meet criteria, you name it.

* Updated media finders to respect format filters; Added credo file
2024-02-09 13:23:37 -08:00
Kieran 89497c4c6b First UI Pass (#15)
* Added scratchpad dir

* Installed Alpine

* [WIP] began integrating Tailwind and accompanying theme

* [WIP] Set up basic views for sources

* Adds new UI to media profiles page

* Removes unneeded view
2024-02-08 11:03:11 -08:00
Kieran 013b2ccc21 Media downloading indexing options (round 2) (#14)
* Adds options + option builder + metadata parsing for media thumbnails

* Added release-type options to media profile; built option parser for indexing operations

* Added new media_profile options to creation form; made show helper for rendering database items

* Added options for downloading/embedding metadata

* Adds option on sources to not download media (index only)

* reformatted docs
2024-02-06 15:36:41 -08:00
Kieran ca01f17b58 Start integrating playlist support (#13)
* [WIP] updated the output of VideoCollection to include playlists

* Updated source's name to collection_name; supported playlist ID/name fetching

* Hooked up collection_type to form; refactored enqueue_pending_media_downloads

* Added friendly_name to form

* Added media profile link to source view

* Updates comment
2024-02-05 11:38:19 -08:00
Kieran bdef6c75bb Change Channel to Source (#12)
* Ensure channel detail lookup doesn't download the video - oops

* Ran the needful migrations for replacing channels with sources

* got media source test back working

* channel tasks test

* Media indexing worker test

* media tests

* Got all tests working except controller tests

* got all tests working

* Renamed Channel struct to Source

* renamed ChannelTasks

* More renaming; renamed channel details module

* Removed Channel yt-dlp module, instead throwing things in the VideoCollection module

* Renamed what looks like the last of the outstanding data
2024-02-02 10:45:21 -08:00
Kieran d25e65f822 Allow subtitle downloading (#11)
* Added subtitle options to media profile model

* Updated media profile form

* Adds subtitle-based options in options builder

* Updates metadata parser to include subtitles

* Adds subtitle_filepaths to media_item

* renamed video_filepath to media_filepath

* Added more fields to media profile show page
2024-01-31 10:55:02 -08:00
Kieran a5e7c48f70 Download video(s) (first iteration) (#5)
* Updated package.json (and made an excuse to make a branch)

* Video filepath parser (#6)

* Restructured files; Added parser placeholder

* More restructuring

* Added basic parser for hydrating template strings

* Improved docs

* More docs

* Initial implementation of media profiles (#7)

* [WIP] Added basic video download method

* [WIP] Very-WIP first steps at parsing options and downloading

* Made my options safe by default and removed special safe versions

* Ran html generator for mediaprofile model - leaving as-is for now

* Addressed a bunch of TODO comments

* Add "channel" type Media Source (#8)

* [WIP] Working on fetching channel metadata in yt-dlp backend

* Finished first draft of methods to do with querying channels

* Renamed CommandRunnerMock to have a more descriptive name

* Ran the phx generator for the channel model

* Renamed Downloader namespace to MediaClient

* [WIP] saving before attempting LiveView

* LiveView did not work out but here's a working controller how about

* Index a channel (#9)

* Ran a MediaItem generator; Reformatted to my liking

* [WIP] added basic index function

* setup oban

* Added basic Oban job for indexing

* Added in workers for indexing; hooked them into record creation flow

* Added a task model with a phx generator

* Tied together tasks with jobs and channels

* Download indexed videos (#10)

* Clarified documentation

* more comments

* [WIP] hooked up basic video downloading; starting work on metadata

* Added metadata model and parsing

Adding the metadata model made me realize that, in many cases, yt-dlp
returns undesired input in stdout, breaking parsing. In order to get
the metadata model working, I had to change the way in which the app
interacts with yt-dlp. Now, output is written as a file to disk which
is immediately re-read and returned.

* Added tests for video download worker

* Hooked up video downloading to the channel indexing pipeline

* Adds tasks for media items

* Updated video metadata parser to extract the title

* Ran linting
2024-01-30 20:15:59 -08:00
Kieran fee39151a8 Add CI (#4)
* Added credo

* Added and ran Prettier; First pass at GH Actions

* Updated actions workflow to hopefully work on this branch

* Name the folder correctly how about

* Added proper container prefix to commands

* Apparently you have to get back into the root after commands

* CI is working, update branches to final value
2024-01-19 23:43:25 -08:00
Kieran 4827da3106 Initial yt-dlp implementation (#3)
* Added basic runner for yt-dlp commands

* Updated formatting rules

* Added JSON runner; Added test script for CLI interactions

* Reworked yt-dlp runner; added proper test mocks
2024-01-19 22:26:32 -08:00
Kieran 536f868ba0 Initial setup (first pass) (#1)
* Updated Elixir version; made phx interactive

* Added JS formatting rules

* Added yt-dlp to dockerfile

* Added a basic PR template

* Added default responses for the PR template
2024-01-19 08:49:16 -08:00
9 changed files with 31 additions and 39 deletions
-1
View File
@@ -51,7 +51,6 @@ If it doesn't work for your use case, please make a feature request! You can als
- Advanced options like setting cutoff dates and filtering by title
- Reliable hands-off operation
- Can pass cookies to YouTube to download your private playlists ([docs](https://github.com/kieraneglin/pinchflat/wiki/YouTube-Cookies))
- Sponsorblock integration
## Screenshots
+2 -1
View File
@@ -32,7 +32,8 @@ config :pinchflat, Pinchflat.Mailer, adapter: Swoosh.Adapters.Test
# Disable swoosh api client as it is only required for production adapters.
config :swoosh, :api_client, false
config :logger, level: :critical
# Print only warnings and errors during test
config :logger, level: :warning
# Initialize plugs at runtime for faster test compilation
config :phoenix, :plug_init_mode, :runtime
@@ -57,9 +57,7 @@ defmodule Pinchflat.Downloading.MediaDownloadWorker do
{:ok, updated_media_item}
err ->
Logger.error("Failed to download media for media item #{media_item.id}: #{inspect(err)}")
{:error, :download_failed}
err
end
end
+6 -6
View File
@@ -43,25 +43,25 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do
xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>#{safe(source.custom_name)}</title>
<link>#{safe(source.original_url)}</link>
<link>#{source.original_url}</link>
<description>#{safe(source.description)}</description>
<category>TV &amp; Film</category>
<generator>Generated by Pinchflat</generator>
<language>en-us</language>
<lastBuildDate>#{Calendar.strftime(source.updated_at, @datetime_format)}</lastBuildDate>
<pubDate>#{Calendar.strftime(source.inserted_at, @datetime_format)}</pubDate>
<atom:link href="#{safe(generate_self_link(url_base, source))}" rel="self" type="application/rss+xml" />
<atom:link href="#{generate_self_link(url_base, source)}" rel="self" type="application/rss+xml" />
<podcast:locked>yes</podcast:locked>
<podcast:guid>#{source.uuid}</podcast:guid>
<image>
<url>#{safe(feed_image_path)}</url>
<url>#{feed_image_path}</url>
<title>#{safe(source.custom_name)}</title>
<link>#{safe(source.original_url)}</link>
<link>#{source.original_url}</link>
</image>
<itunes:author>#{safe(source.custom_name)}</itunes:author>
<itunes:subtitle>#{safe(source.custom_name)}</itunes:subtitle>
<itunes:block>yes</itunes:block>
<itunes:image href="#{safe(feed_image_path)}"></itunes:image>
<itunes:image href="#{feed_image_path}"></itunes:image>
<itunes:explicit>false</itunes:explicit>
<itunes:category text="TV &amp; Film"></itunes:category>
@@ -77,7 +77,7 @@ defmodule Pinchflat.Podcasts.RssFeedBuilder do
<item>
<guid isPermaLink="false">#{media_item.uuid}</guid>
<title>#{safe(media_item.title)}</title>
<link>#{safe(media_item.original_url)}</link>
<link>#{media_item.original_url}</link>
<description>#{safe(media_item.description)}</description>
<pubDate>#{generate_upload_date(media_item)}</pubDate>
<itunes:duration>#{media_item.duration_seconds}</itunes:duration>
@@ -1,27 +1,27 @@
<div class="grid grid-cols-1 gap-4 md:grid-cols-3">
<div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
<a href={~p"/media_profiles"} class="mt-4 flex flex-col items-center justify-center">
<div class="mt-4 flex flex-col items-center justify-center">
<span class="text-md font-medium">Media Profile(s)</span>
<h4 class="text-title-md font-bold text-white">
<%= @media_profile_count %>
</h4>
</a>
</div>
</div>
<div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
<a href={~p"/sources"} class="mt-4 flex flex-col items-center justify-center">
<div class="mt-4 flex flex-col items-center justify-center">
<span class="text-md font-medium">Source(s)</span>
<h4 class="text-title-md font-bold text-white">
<%= @source_count %>
</h4>
</a>
</div>
</div>
<div class="rounded-sm border px-7.5 py-6 shadow-default border-strokedark bg-boxdark">
<a href="#" class="mt-4 flex flex-col items-center justify-center">
<div class="mt-4 flex flex-col items-center justify-center">
<span class="text-md font-medium">Media Item(s)</span>
<h4 class="text-title-md font-bold text-white">
<%= @media_item_count %>
</h4>
</a>
</div>
</div>
<span class="text-strokedark">I know this page isn't super useful yet, but give it time :&rpar;</span>
</div>
+14 -11
View File
@@ -1,6 +1,5 @@
defmodule PinchflatWeb.Router do
use PinchflatWeb, :router
import Phoenix.LiveDashboard.Router
# IMPORTANT: `strip_trailing_extension` in endpoint.ex removes
# the extension from the path
@@ -12,7 +11,6 @@ defmodule PinchflatWeb.Router do
plug :put_root_layout, html: {PinchflatWeb.Layouts, :root}
plug :protect_from_forgery
plug :put_secure_browser_headers
plug :allow_iframe_embed
end
pipeline :api do
@@ -47,12 +45,21 @@ defmodule PinchflatWeb.Router do
get "/media/:uuid/stream", MediaItems.MediaItemController, :stream
end
scope "/dev" do
pipe_through :browser
# Enable LiveDashboard and Swoosh mailbox preview in development
if Application.compile_env(:pinchflat, :dev_routes) do
# If you want to use the LiveDashboard in production, you should put
# it behind authentication and allow only admins to access it.
# If your application does not have an admins-only section yet,
# you can use Plug.BasicAuth to set up some basic authentication
# as long as you are also using SSL (which you should anyway).
import Phoenix.LiveDashboard.Router
live_dashboard "/dashboard",
metrics: PinchflatWeb.Telemetry,
ecto_repos: [Pinchflat.Repo]
scope "/dev" do
pipe_through :browser
live_dashboard "/dashboard", metrics: PinchflatWeb.Telemetry
forward "/mailbox", Plug.Swoosh.MailboxPreview
end
end
defp maybe_basic_auth(conn, opts) do
@@ -77,8 +84,4 @@ defmodule PinchflatWeb.Router do
defp credential_set?(credential) do
credential && credential != ""
end
defp allow_iframe_embed(conn, _opts) do
delete_resp_header(conn, "x-frame-options")
end
end
+2 -3
View File
@@ -4,7 +4,7 @@ defmodule Pinchflat.MixProject do
def project do
[
app: :pinchflat,
version: "0.1.6",
version: "0.1.5",
elixir: "~> 1.16",
elixirc_paths: elixirc_paths(Mix.env()),
start_permanent: Mix.env() == :prod,
@@ -33,7 +33,7 @@ defmodule Pinchflat.MixProject do
def application do
[
mod: {Pinchflat.Application, []},
extra_applications: [:logger, :runtime_tools, :inets, :os_mon]
extra_applications: [:logger, :runtime_tools, :inets]
]
end
@@ -50,7 +50,6 @@ defmodule Pinchflat.MixProject do
{:phoenix_ecto, "~> 4.4"},
{:ecto_sql, "~> 3.10"},
{:ecto_sqlite3, ">= 0.0.0"},
{:ecto_sqlite3_extras, "~> 1.2.0"},
{:phoenix_html, "~> 3.3"},
{:phoenix_live_reload, "~> 1.2", only: :dev},
{:phoenix_live_view, "~> 0.20.1"},
-2
View File
@@ -13,7 +13,6 @@
"ecto": {:hex, :ecto, "3.11.1", "4b4972b717e7ca83d30121b12998f5fcdc62ba0ed4f20fd390f16f3270d85c3e", [:mix], [{:decimal, "~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "ebd3d3772cd0dfcd8d772659e41ed527c28b2a8bde4b00fe03e0463da0f1983b"},
"ecto_sql": {:hex, :ecto_sql, "3.11.1", "e9abf28ae27ef3916b43545f9578b4750956ccea444853606472089e7d169470", [:mix], [{:db_connection, "~> 2.4.1 or ~> 2.5", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.11.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:myxql, "~> 0.6.0", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.16.0 or ~> 0.17.0 or ~> 1.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:tds, "~> 2.1.1 or ~> 2.2", [hex: :tds, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.0 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "ce14063ab3514424276e7e360108ad6c2308f6d88164a076aac8a387e1fea634"},
"ecto_sqlite3": {:hex, :ecto_sqlite3, "0.15.1", "40f2fbd9e246455f8c42e7e0a77009ef806caa1b3ce6f717b2a0a80e8432fcfd", [:mix], [{:decimal, "~> 1.6 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:ecto, "~> 3.11", [hex: :ecto, repo: "hexpm", optional: false]}, {:ecto_sql, "~> 3.11", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:exqlite, "~> 0.19", [hex: :exqlite, repo: "hexpm", optional: false]}], "hexpm", "28b16e177123c688948357176662bf9ff9084daddf950ef5b6baf3ee93707064"},
"ecto_sqlite3_extras": {:hex, :ecto_sqlite3_extras, "1.2.2", "36e60b561a11441d15f26c791817999269fb578b985162207ebb08b04ca71e40", [:mix], [{:exqlite, ">= 0.13.2", [hex: :exqlite, repo: "hexpm", optional: false]}, {:table_rex, "~> 4.0", [hex: :table_rex, repo: "hexpm", optional: false]}], "hexpm", "2b66ba7246bb4f7e39e2578acd4a0e4e4be54f60ff52d450a01be95eeb78ff1e"},
"elixir_make": {:hex, :elixir_make, "0.7.8", "505026f266552ee5aabca0b9f9c229cbb496c689537c9f922f3eb5431157efc7", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:certifi, "~> 2.0", [hex: :certifi, repo: "hexpm", optional: true]}], "hexpm", "7a71945b913d37ea89b06966e1342c85cfe549b15e6d6d081e8081c493062c07"},
"esbuild": {:hex, :esbuild, "0.8.1", "0cbf919f0eccb136d2eeef0df49c4acf55336de864e63594adcea3814f3edf41", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "25fc876a67c13cb0a776e7b5d7974851556baeda2085296c14ab48555ea7560f"},
"ex_check": {:hex, :ex_check, "0.14.0", "d6fbe0bcc51cf38fea276f5bc2af0c9ae0a2bb059f602f8de88709421dae4f0e", [:mix], [], "hexpm", "8a602e98c66e6a4be3a639321f1f545292042f290f91fa942a285888c6868af0"},
@@ -47,7 +46,6 @@
"ranch": {:hex, :ranch, "1.8.0", "8c7a100a139fd57f17327b6413e4167ac559fbc04ca7448e9be9057311597a1d", [:make, :rebar3], [], "hexpm", "49fbcfd3682fab1f5d109351b61257676da1a2fdbe295904176d5e521a2ddfe5"},
"sobelow": {:hex, :sobelow, "0.13.0", "218afe9075904793f5c64b8837cc356e493d88fddde126a463839351870b8d1e", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "cd6e9026b85fc35d7529da14f95e85a078d9dd1907a9097b3ba6ac7ebbe34a0d"},
"swoosh": {:hex, :swoosh, "1.14.4", "94e9dba91f7695a10f49b0172c4a4cb658ef24abef7e8140394521b7f3bbb2d4", [:mix], [{:cowboy, "~> 1.1 or ~> 2.4", [hex: :cowboy, repo: "hexpm", optional: true]}, {:ex_aws, "~> 2.1", [hex: :ex_aws, repo: "hexpm", optional: true]}, {:finch, "~> 0.6", [hex: :finch, repo: "hexpm", optional: true]}, {:gen_smtp, "~> 0.13 or ~> 1.0", [hex: :gen_smtp, repo: "hexpm", optional: true]}, {:hackney, "~> 1.9", [hex: :hackney, repo: "hexpm", optional: true]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mail, "~> 0.2", [hex: :mail, repo: "hexpm", optional: true]}, {:mime, "~> 1.1 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug, "~> 1.9", [hex: :plug, repo: "hexpm", optional: true]}, {:plug_cowboy, ">= 1.0.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}, {:req, "~> 0.4 or ~> 1.0", [hex: :req, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.2 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "081c5a590e4ba85cc89baddf7b2beecf6c13f7f84a958f1cd969290815f0f026"},
"table_rex": {:hex, :table_rex, "4.0.0", "3c613a68ebdc6d4d1e731bc973c233500974ec3993c99fcdabb210407b90959b", [:mix], [], "hexpm", "c35c4d5612ca49ebb0344ea10387da4d2afe278387d4019e4d8111e815df8f55"},
"tailwind": {:hex, :tailwind, "0.2.2", "9e27288b568ede1d88517e8c61259bc214a12d7eed271e102db4c93fcca9b2cd", [:mix], [{:castore, ">= 0.0.0", [hex: :castore, repo: "hexpm", optional: false]}], "hexpm", "ccfb5025179ea307f7f899d1bb3905cd0ac9f687ed77feebc8f67bdca78565c4"},
"telemetry": {:hex, :telemetry, "1.2.1", "68fdfe8d8f05a8428483a97d7aab2f268aaff24b49e0f599faa091f1d4e7f61c", [:rebar3], [], "hexpm", "dad9ce9d8effc621708f99eac538ef1cbe05d6a874dd741de2e689c47feafed5"},
"telemetry_metrics": {:hex, :telemetry_metrics, "0.6.2", "2caabe9344ec17eafe5403304771c3539f3b6e2f7fb6a6f602558c825d0d0bfb", [:mix], [{:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "9b43db0dc33863930b9ef9d27137e78974756f5f198cae18409970ed6fa5b561"},
@@ -74,12 +74,6 @@ defmodule Pinchflat.Downloading.MediaDownloadWorkerTest do
end)
end
test "it ensures error are returned in a 2-item tuple", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, fn _url, _opts, _ot -> {:error, "error", 1} end)
assert {:error, :download_failed} = perform_job(MediaDownloadWorker, %{id: media_item.id})
end
test "it does not download if the source is set to not download", %{media_item: media_item} do
expect(YtDlpRunnerMock, :run, 0, fn _url, _opts, _ot -> :ok end)