aboutsummaryrefslogtreecommitdiffhomepage
path: root/crates/shirabe/src/util/http_downloader.rs
AgeCommit message (Collapse)Author
2026-07-18perf(regex): eliminate per-call clone overhead in preg_* dispatchnsfisis
regex::Regex::clone() does not share the underlying meta engine's search-cache pool, so every fresh clone pays a ~10us warmup cost on its first use. Two changes together eliminate this across nearly all preg_* call sites: - A php_regex! macro resolves PHP-style patterns to a per-call-site &'static regex::Regex (via regex-macro's LazyLock), applied at the majority of call sites throughout the codebase. - Call sites still passing dynamic pattern strings go through PATTERN_CACHE, which now stores Arc<(Regex, bool)> and hands out Arc::clone()s instead of cloning the Regex itself. PregPattern::resolve() returns a ResolvedPattern enum (Arc or 'static reference) rather than an owned Regex, so neither path ever clones the Regex proper. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18refactor(curl-downloader): drop unnecessary Rc wrapping around handlesnsfisis
CurlDownloader::auth_helper was never cloned out to another owner, so Rc<RefCell<AuthHelper>> only needed the RefCell for interior mutability (all methods take &self). Likewise HttpDownloader::dispatch cloned self.rfs into a local binding it only ever used synchronously (copy/get_contents don't await), so the clone bought nothing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17perf(sync-executor): drive HTTP fetches through one real top-level runtimensfisis
Replace sync_executor::block_on's reactor-less busy-spin poller with tokio::task::block_in_place + Handle::current().block_on(), riding a single tokio Runtime entered once in main.rs (falling back to a disposable one when no ambient runtime exists, e.g. in tests). This lets HttpDownloader::dispatch await CurlDownloader::download directly instead of bouncing through the separate curl_runtime() bridge, which is now deleted. Manual create-project verification against the real network caught a concurrency bug this exposed: async_fetch_file held http_downloader's RefMut across the await on add(), which only panics once downloads genuinely overlap. add() only needs &self, so borrow() fixes it. With everything now sharing one real reactor, the FuturesOrdered fan-out added for ComposerRepository::get_security_advisories/ load_async_packages finally overlaps for real: fetching 8 packages' metadata dropped from ~7-40s to a consistent ~3-4s in a before/after comparison, with identical resulting lock files. sync_executor::block_on's call sites are still synchronous rather than async fn propagated up to Command::execute, which remains the end goal (see the TODO(phase-e) in sync_executor.rs) - nested block_on calls elsewhere don't get this same overlap, only prevented panics.
2026-07-17refactor(http-downloader): drop the job table for a &self Semaphore corensfisis
Replaces Job/Request/JobHandle/id_gen/running_jobs/max_jobs with a tokio::sync::Semaphore permit held for the duration of each request. get/add/copy/add_copy are now &self (add/add_copy are also genuinely async); a shared execute()/dispatch() core replaces add_job/run_rfs_job/start_job/settle_job, returning the Response directly instead of deferring to wait()/count_active_jobs()/ get_response() (all removed — confirmed zero callers, same for the now-unused STATUS_* constants). get()/copy() stay synchronous rather than becoming async wrappers around add()/add_copy(), bridging via the existing sync_executor instead of the curl_runtime() introduced for CurlDownloader: their callers (~35 files reaching HttpDownloader) are mostly plain sync fns with no async boundary anywhere in the call chain, and forcing that propagation now would pull forward the dedicated async-propagation task. curl-eligible requests still route through curl_runtime() inside dispatch(), same as before — nesting sync_executor::block_on (no real reactor) around curl_runtime().block_on() (a real, separate Runtime) is safe; it's only nesting curl_runtime() inside itself that would panic. CurlDownloader no longer needs Rc<RefCell<>> wrapping despite the original design sketch: since item 2 made all of its methods &self, a plain Option<CurlDownloader> field works fine under HttpDownloader's own &self methods. get/add/copy/add_copy becoming &self (rather than &mut self) requires no changes at any of their ~35 calling files: RefMut/Ref both deref to a type that can call &self methods just fine. Verified manually against real network I/O (sandbox disabled): `shirabe show -a` (get()'s sync_executor-bridged path) and `shirabe create-project` (add_copy()'s genuinely async path via file_downloader.rs) both complete correctly with no hang.
2026-07-17refactor(curl-downloader): rewrite as a single async fn, drop Job/ticknsfisis
Replaces the Job-table + tick()-driven polling loop with one async download() that sends, decides (retry/redirect/fail/succeed via a new decide() extracted from the former run_job), and loops until it resolves — no more resolve/reject callbacks. The client switches from reqwest::blocking::Client to the non-blocking reqwest::Client, with body streaming now via tokio::fs. Because real async I/O needs a live tokio reactor and none runs yet at the process level (sync_executor::block_on is a no-reactor busy-spin executor that only works when awaited futures resolve synchronously), HttpDownloader::start_job drives CurlDownloader::download() through a dedicated temporary current_thread Runtime (curl_runtime(), marked TODO(phase-e)) instead. This keeps concurrency characteristics unchanged for now — start_job still resolves one job at a time — real parallel I/O lands once HttpDownloader/Loop are rearchitected on top of FuturesUnordered. count_active_jobs' curl.tick() polling and the Job.settled/curl_id plumbing are removed as dead weight now that start_job settles curl jobs synchronously, same as the rfs path already did. abort_request is dropped: it had no caller (the PHP Promise-cancellation flow it backs was never ported), and the job table it operated on no longer exists. Verified manually against real network I/O (sandbox disabled): `shirabe show -a` (JSON metadata, in-memory body) and `shirabe create-project` (actual dist zip download + extraction) both complete correctly with no hang. Two unrelated pre-existing bugs surfaced during manual testing (an event-dispatcher subscriber wiring gap during `require`, and a RefCell reentrancy panic in `diagnose`) reproduce identically on the pre-change code and are out of scope here.
2026-07-16refactor(http-downloader): wrap HttpDownloaderMockState in Rc<RefCell<>>nsfisis
In prep for the upcoming &self conversion of add()/get()/copy(), the mock hook needs interior mutability too. The struct's Clone derive is dropped since nothing clones the whole state anymore, only the shared Rc handle.
2026-07-16refactor(remote-filesystem): return headers from copy/get_contentsnsfisis
Wrap RemoteFilesystem in Rc<RefCell<>> inside HttpDownloader, in prep for the upcoming &self conversion of add()/get(). copy()/get_contents() now bundle the response headers into their return value instead of requiring a follow-up get_last_headers() call, since two separate calls through a shared RefCell could otherwise race: nothing would guarantee the reader observes the headers from its own request rather than one clobbered by a concurrently borrowed call. get_last_headers() itself is left in place, mirroring RemoteFilesystem::getLastHeaders() in PHP.
2026-06-29chore(lint): ban bare `use anyhow::Result` and fully qualify itnsfisis
Add a no_banned_use linter that forbids importing anyhow::Result, and update all call sites to reference it via its fully-qualified path so it is never confused with std::result::Result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28refactor: add linternsfisis
2026-06-27refactor: fix compiler warnings and clippy warningsnsfisis
2026-06-26feat(http): reimplement CurlDownloader on reqwest; port 15 more testsnsfisis
Replace the libcurl-shim CurlDownloader with a reqwest+tokio implementation per the .ken sketch, resolving the construction panic that blocked command tests (mock path via __new_mock is untouched). Port remote_filesystem (7), hg/svn driver (4), zip_archiver/git_exclude_filter (4) tests. Fix hg/svn/git_exclude regex-delimiter and svn result-propagation porting bugs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26test: port 44 vcs/downloader/version tests using mock infransfisis
Port git, version_guesser, gitlab_driver, github_driver, and git_downloader tests using the ProcessExecutor/HttpDownloader mocks and IO/Config stubs. Fix production regex-porting bugs surfaced by the now-reachable paths: Url::sanitize and Response::find_header_value had non-delimited PCRE patterns; implement array_search_mixed non-strict branch and a datetime format mapping. Add HttpDownloader::__new_mock so mocked downloaders skip curl construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26feat(test): add HttpDownloaderMock, IOStub, and Config stub helpersnsfisis
IOStub and ConfigStubBuilder provide getMockBuilder-style configurable stubs. Wired into util/repository/downloader/command test targets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21refactor(math): use method-style max/min/clamp over std::cmpnsfisis
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20refactor(php-shim): drop Box wrapping from PhpMixed List/Arraynsfisis
The List and Array variants of PhpMixed boxed their elements unnecessarily. Store PhpMixed values directly and update all callers accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20refactor: auto-fix clippy warningsnsfisis
2026-06-14refactor(pcre): drop Result from Preg method return typesnsfisis
The Preg methods panic on PCRE failure (per the file header rationale), so their anyhow::Result wrappers never carried an Err. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14refactor(pcre): drop strict-groups Preg variantsnsfisis
Rust's type system already distinguishes participating from non-participating capture groups via Option, so the *StrictGroups methods add no safety here. Remove them and switch callers to the plain variants.
2026-06-10feat(phase-c): resolve cross-module phase-b TODOsnsfisis
2026-06-06refactor(http-response): take url directly instead of request mapnsfisis
Response only ever reads the url out of the request array, so accept it as a String directly. With url always present the 'url key missing' LogicException can no longer fire, so Response::new and CurlResponse::new return Self instead of a double Result. Also drops the unused from_php_mixed/to_php_mixed stubs and the request_to_map helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04feat: resolve trivial todo!() sites with existing constructorsnsfisis
Replace todo!("VersionParser::new()") with VersionParser::new() at 6 call sites (array_loader, base_command, create_project_command, vcs_repository, http_downloader x2) and EventDispatcher::io_clone with self.io.clone(). All targets already exist on their types. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03feat(downloader,repository): wire exception instanceof downcasts via anyhownsfisis
Resolve the PHP try/catch instanceof checks that select on exception type: TransportException-only catches in RepositorySet, SvnDriver and JsonFile; the RuntimeException/IrrecoverableDownloadException handleError closure and the RuntimeException update-retry catch in DownloadManager. Each uses anyhow::Error::downcast_ref and reads the exception's message field directly (Display is todo\!() on the shim exceptions). PHPUnit\Framework\Exception checks in VcsDownloader are documented as intentionally always-false since the test framework is out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29refactor(io): unify IOInterface params to Rc<RefCell<dyn _>>nsfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26refactor(io): share IOInterface via Rc<RefCell<dyn _>> handlensfisis
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23refactor(semver): change ConstraintInterface to a closed enumnsfisis
Replace the dyn ConstraintInterface trait objects with an AnyConstraint enum closing over its four implementors (Simple, Multi, MatchAll, MatchNone), mirroring the earlier Rule enum conversion. Rename constraint.rs to simple_constraint.rs to match the renamed Constraint type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23refactor(promise): drop \React\Promisensfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23refactor(promise): change functions returning PromiseInterface to async fnnsfisis
2026-05-22refactor: share Pool via Rc<RefCell>nsfisis
Convert Pool to Rc<RefCell<Pool>> so Solver, Decisions, and RuleSetGenerator share it, resolving the todo!() placeholders that blocked the dependency resolver (Phase C shared ownership). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22refactor(composer): unify Composer/PartialComposer via Rc handlesnsfisis
Model PHP's `Composer extends PartialComposer` as a PartialOrFullComposer enum and merge partial_composer.rs into composer.rs. Introduce ComposerHandle / PartialComposerHandle (plus their Weak variants) so the graph can be shared, and build it at once with Rc::new_cyclic in the factory to resolve the back-reference cycles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20refactor: re-export module items to shorten import pathsnsfisis
2026-05-20fix(compile): fix all remaining compile errorsnsfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19fix(compile): fix more random compile errorsnsfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19fix(compile): fix various compile errorsnsfisis
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-17fix(shirabe): correct invalid import paths and add missing dependencynsfisis
2026-05-17fix(php-shim): add missing stub functions, constants, and typesnsfisis
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-17chore(style): cargo fmtnsfisis
2026-05-16feat(port): port HttpDownloader.phpnsfisis
2026-05-12feat(init): add scaffold filesnsfisis