aboutsummaryrefslogtreecommitdiffhomepage
path: root/crates/shirabe/src/util/http_downloader.rs
AgeCommit message (Collapse)Author
2026-08-24refactor(silencer): stop guarding work that stays inside Rustnsfisis
Silencer only lowers the PHP error_reporting() level and re-throws whatever the guarded work raises. A region that never reaches the PHP runtime has no level to lower and emits no diagnostic on failure, so wrapping it is indistinguishable from running it unguarded. The pair kept in Application::hint_common_errors brackets a getComposer() call, which loads installed plugins and dispatches PluginEvents::INIT. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23fix(fs): carry file_get_contents results as bytesnsfisis
file_get_contents() and file_get_contents_with_max_length() return Vec<u8> instead of a from_utf8_lossy'd String. Call sites whose consumer takes a &str still convert lossily and are marked TODO(bytes). file_get_contents_with_max_length() now reads at most the requested number of bytes instead of the whole file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18refactor(function-exists): drop checks for always-present capabilitiesnsfisis
function_exists() returns false in PHP when a function is blocked by disable_functions, when its extension is not compiled in, or when the PHP version predates it. None of those apply to a native binary.
2026-08-18refactor(preg): add preg_is_match for existence-only call sitesnsfisis
The capture groups were discarded at 162 of the preg_match call sites, which only tested the Option. They now call preg_is_match, which lets the regex engine skip capture tracking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18refactor(preg): drop the offset argument from preg_matchnsfisis
Every call site but one passed offset 0. The remaining one, the UTF-8 chunking loop in Application, slices the subject instead: its pattern has no anchor or lookaround, so matching a suffix is equivalent to starting the search at that offset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18refactor(pcre): inline Preg into its call sites and drop the cratensfisis
Preg had shed everything it owned: after the last few rounds its methods were one-line forwards to the shim's preg_*(), differing only in a default argument or a wrapper the caller unwrapped anyway. The 460 call sites now name the shim function, and shirabe-pcre is gone from the workspace along with its LICENSE entry. The forwards expand as they read: isMatch becomes preg_match2(.., 0).is_some() (is_none() where PHP negates it), isMatch3 and match3 drop the .is_some(), matchAll counts through preg_match_all2(..).occurrence_count(), and replace4/replace5 spell out the limit and count arguments preg_replace2 takes. Callbacks are the one place the shapes differ: preg_replace_callback carries an error out of the callback, so the fourteen infallible closures wrap their result in Ok() and expect() it back. Config::process() is the fifteenth, and it drops the `error` cell it captured to smuggle a failure past a closure that could only return a String. The `?` in the closure now carries it, which is what the PHP does -- a throw from the callback leaves preg_replace_callback at the failing match rather than running the remaining replacements and reporting the last error. The module doc that explained why composer/pcre's exceptions and *StrictGroups() variants have no counterpart moves to the shim's preg module, where the functions it describes live. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18refactor(preg): split PregMatches reads into get() and name()nsfisis
PregMatches keyed both forms of a capture group through CaptureKey, so every read built one: a usize wrapped in an enum, or worse, a String allocated to name a group that regex::Captures can look up from a &str. It now mirrors regex::Captures instead -- get() takes the group number, name() the group name -- and the enum drops out of the type entirely. That is 285 call sites across 59 files, and the named ones carry most of the win: `matches.get(&CaptureKey::ByName("host".to_string()))` reads as `matches.name("host")`. ProcessExecutor loses a `user_key` binding that existed only to build the key once. CaptureKey stays as the key type of PregMatchesAll and PregMatchesAllWithOffsets, where numbered and named entries share one IndexMap and a key type is the point. Five files still name it. Also retargets the two preg_match_all comments that described the occurrence count through `matches[&CaptureKey::ByIndex(0)].len()`, an Index impl these types no longer carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18refactor(pcre): hand back the match instead of copying it outnsfisis
Preg::match4 and Preg::replace_callback gave callers a PregMatchedGroups: an IndexMap rebuilt from the match with an owned String per group, plus a second String for a named group's name key. That is the copy PregMatches shed when it started wrapping regex::Captures, reinstated one layer up -- and nearly every regex call in the tree goes through Preg rather than the shim's preg_* directly, so almost nothing saw the borrow. PregMatchedGroups existed only to drop the null (unmatched) groups the old PregMatches held as Option<String> values. PregMatches::get reports a non-participating group as None on its own, so the two read alike and the type collapses into it. Call sites still reach groups through get(&CaptureKey::ByIndex(N)); what changes is that the value arrives as a &str borrowed from the subject, which the signatures now carry as a lifetime. Three places needed the borrow reckoned with rather than a mechanical rewrite: PhpFileCleaner::clean and Problem::get_messages read their groups out before mutating what the match borrows, and Git::get_authentication_failure names the lifetime of its url argument, which the result borrows instead of self. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17refactor(pcre): return the Preg $matches instead of filling an out-paramnsfisis
`Composer\Pcre\Preg` fills `$matches` through a by-ref parameter, and the port mirrored that with a `&mut` (or `Option<&mut>`) out-param plus a bool or count return. Callers had to declare an empty map one line ahead of the call, and the type never said the map is only meaningful when the call matched. Return the matches instead: - match3/match4/is_match3/is_match4 -> Option<PregMatchedGroups> - is_match_named -> Option<PregNamedGroups> - match_all2/is_match_all -> PregMatchesAll - is_match_all_with_offsets3 -> PregMatchesAllWithOffsets Nothing is lost: the bool is `Option::is_some()`, and the occurrence count is the length of any one column of a PREG_PATTERN_ORDER map, now spelled `PregMatchesAll::occurrence_count()`. is_match() still answers the bool question directly for callers that want no groups. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17refactor(preg): wrap the preg_* $matches maps in newtypesnsfisis
The five IndexMap shapes that the preg_* functions and Preg fill in are now distinct types generated by preg_match_map!, so a matches map no longer interchanges with any other map of the same key and value type. Index<usize> is kept alongside Index<&Q> because call sites such as config_command and event_dispatcher reach for a group by its position in the map rather than by its capture key.
2026-08-12docs(todo): retag TODO markers by root causensfisis
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-09refactor(pcre): extract composer/pcre into the shirabe-pcre cratensfisis
Move `Composer\Pcre` out of shirabe-external-packages and into its own crate, so the path is `shirabe_pcre::preg::Preg` instead of `shirabe_external_packages::composer::pcre::preg::Preg`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08feat(php-shim): give ported exceptions PHP's class hierarchynsfisis
Ported exceptions were flat structs reached with `downcast_ref`, so Composer's `catch (\RuntimeException $e)` only matched the exact leaf type and `get_class($e)` had nothing to report. Each exception now embeds an instance of the class it extends and travels inside an `AnyThrowable`; `Catch::catch`/`catch_mut` walk that chain, and `PhpClass::php_class_name` yields the PHP FQCN. Dropping the `std::error::Error` impls from the exception types leaves `AnyThrowable` as the only route into an `anyhow::Error`, so the walk cannot be bypassed. A `no_exception_downcast` linter catches the `downcast::<X>()` calls that would now silently answer `None`. Three sites change behavior as a result: the `TransportException` exit-code override reaches `MaxFileSizeExceededException`, the `catch (\LogicException)` in findSimilar() reaches its subclasses, and rendered exception titles carry the real class name rather than a guess. `get_class_err()` is no longer a `todo!()`, which re-enables FilesystemRepositoryTest::testCorruptedRepositoryFile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06chore: drop @param/@return tags that only restate Rust typesnsfisis
The ported docblocks copied @param and @return straight from the PHP source. When such a tag carries nothing but a type and an argument name, the Rust signature already states it, so the line is noise. Tags whose text adds prose beyond the type are kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25refactor: replace redundant clones with movesnsfisis
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-20fix(no-proxy-pattern): stop chr() corrupting IP bytes as lossy UTF-8nsfisis
shirabe_php_shim::chr() returned a Rust String, which lossily re-encodes bytes >= 0x80 as UTF-8 replacement characters. ip_get_mask, ip_get_network, and ip_map_to_6 relied on chr() to build raw in_addr and netmask byte arrays, corrupting IPv4-in-IPv6 mappings and CIDR netmasks. Build the Vec<u8> byte arrays directly instead of round-tripping through String, and un-ignore test_ip_address and test_ip_range now that the underlying bug is fixed. chr()'s only other caller (http_downloader.rs, an ASCII ESC byte in a regex pattern) didn't need the indirection either, so remove the shim function entirely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18perf(regex): eliminate per-call clone overhead in preg_* dispatchnsfisis
regex::Regex::clone() does not share the underlying meta engine's search-cache pool, so every fresh clone pays a ~10us warmup cost on its first use. Two changes together eliminate this across nearly all preg_* call sites: - A php_regex! macro resolves PHP-style patterns to a per-call-site &'static regex::Regex (via regex-macro's LazyLock), applied at the majority of call sites throughout the codebase. - Call sites still passing dynamic pattern strings go through PATTERN_CACHE, which now stores Arc<(Regex, bool)> and hands out Arc::clone()s instead of cloning the Regex itself. PregPattern::resolve() returns a ResolvedPattern enum (Arc or 'static reference) rather than an owned Regex, so neither path ever clones the Regex proper. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-18refactor(curl-downloader): drop unnecessary Rc wrapping around handlesnsfisis
CurlDownloader::auth_helper was never cloned out to another owner, so Rc<RefCell<AuthHelper>> only needed the RefCell for interior mutability (all methods take &self). Likewise HttpDownloader::dispatch cloned self.rfs into a local binding it only ever used synchronously (copy/get_contents don't await), so the clone bought nothing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17perf(sync-executor): drive HTTP fetches through one real top-level runtimensfisis
Replace sync_executor::block_on's reactor-less busy-spin poller with tokio::task::block_in_place + Handle::current().block_on(), riding a single tokio Runtime entered once in main.rs (falling back to a disposable one when no ambient runtime exists, e.g. in tests). This lets HttpDownloader::dispatch await CurlDownloader::download directly instead of bouncing through the separate curl_runtime() bridge, which is now deleted. Manual create-project verification against the real network caught a concurrency bug this exposed: async_fetch_file held http_downloader's RefMut across the await on add(), which only panics once downloads genuinely overlap. add() only needs &self, so borrow() fixes it. With everything now sharing one real reactor, the FuturesOrdered fan-out added for ComposerRepository::get_security_advisories/ load_async_packages finally overlaps for real: fetching 8 packages' metadata dropped from ~7-40s to a consistent ~3-4s in a before/after comparison, with identical resulting lock files. sync_executor::block_on's call sites are still synchronous rather than async fn propagated up to Command::execute, which remains the end goal (see the TODO(phase-e) in sync_executor.rs) - nested block_on calls elsewhere don't get this same overlap, only prevented panics.
2026-07-17refactor(http-downloader): drop the job table for a &self Semaphore corensfisis
Replaces Job/Request/JobHandle/id_gen/running_jobs/max_jobs with a tokio::sync::Semaphore permit held for the duration of each request. get/add/copy/add_copy are now &self (add/add_copy are also genuinely async); a shared execute()/dispatch() core replaces add_job/run_rfs_job/start_job/settle_job, returning the Response directly instead of deferring to wait()/count_active_jobs()/ get_response() (all removed — confirmed zero callers, same for the now-unused STATUS_* constants). get()/copy() stay synchronous rather than becoming async wrappers around add()/add_copy(), bridging via the existing sync_executor instead of the curl_runtime() introduced for CurlDownloader: their callers (~35 files reaching HttpDownloader) are mostly plain sync fns with no async boundary anywhere in the call chain, and forcing that propagation now would pull forward the dedicated async-propagation task. curl-eligible requests still route through curl_runtime() inside dispatch(), same as before — nesting sync_executor::block_on (no real reactor) around curl_runtime().block_on() (a real, separate Runtime) is safe; it's only nesting curl_runtime() inside itself that would panic. CurlDownloader no longer needs Rc<RefCell<>> wrapping despite the original design sketch: since item 2 made all of its methods &self, a plain Option<CurlDownloader> field works fine under HttpDownloader's own &self methods. get/add/copy/add_copy becoming &self (rather than &mut self) requires no changes at any of their ~35 calling files: RefMut/Ref both deref to a type that can call &self methods just fine. Verified manually against real network I/O (sandbox disabled): `shirabe show -a` (get()'s sync_executor-bridged path) and `shirabe create-project` (add_copy()'s genuinely async path via file_downloader.rs) both complete correctly with no hang.
2026-07-17refactor(curl-downloader): rewrite as a single async fn, drop Job/ticknsfisis
Replaces the Job-table + tick()-driven polling loop with one async download() that sends, decides (retry/redirect/fail/succeed via a new decide() extracted from the former run_job), and loops until it resolves — no more resolve/reject callbacks. The client switches from reqwest::blocking::Client to the non-blocking reqwest::Client, with body streaming now via tokio::fs. Because real async I/O needs a live tokio reactor and none runs yet at the process level (sync_executor::block_on is a no-reactor busy-spin executor that only works when awaited futures resolve synchronously), HttpDownloader::start_job drives CurlDownloader::download() through a dedicated temporary current_thread Runtime (curl_runtime(), marked TODO(phase-e)) instead. This keeps concurrency characteristics unchanged for now — start_job still resolves one job at a time — real parallel I/O lands once HttpDownloader/Loop are rearchitected on top of FuturesUnordered. count_active_jobs' curl.tick() polling and the Job.settled/curl_id plumbing are removed as dead weight now that start_job settles curl jobs synchronously, same as the rfs path already did. abort_request is dropped: it had no caller (the PHP Promise-cancellation flow it backs was never ported), and the job table it operated on no longer exists. Verified manually against real network I/O (sandbox disabled): `shirabe show -a` (JSON metadata, in-memory body) and `shirabe create-project` (actual dist zip download + extraction) both complete correctly with no hang. Two unrelated pre-existing bugs surfaced during manual testing (an event-dispatcher subscriber wiring gap during `require`, and a RefCell reentrancy panic in `diagnose`) reproduce identically on the pre-change code and are out of scope here.
2026-07-16refactor(http-downloader): wrap HttpDownloaderMockState in Rc<RefCell<>>nsfisis
In prep for the upcoming &self conversion of add()/get()/copy(), the mock hook needs interior mutability too. The struct's Clone derive is dropped since nothing clones the whole state anymore, only the shared Rc handle.
2026-07-16refactor(remote-filesystem): return headers from copy/get_contentsnsfisis
Wrap RemoteFilesystem in Rc<RefCell<>> inside HttpDownloader, in prep for the upcoming &self conversion of add()/get(). copy()/get_contents() now bundle the response headers into their return value instead of requiring a follow-up get_last_headers() call, since two separate calls through a shared RefCell could otherwise race: nothing would guarantee the reader observes the headers from its own request rather than one clobbered by a concurrently borrowed call. get_last_headers() itself is left in place, mirroring RemoteFilesystem::getLastHeaders() in PHP.
2026-06-29chore(lint): ban bare `use anyhow::Result` and fully qualify itnsfisis
Add a no_banned_use linter that forbids importing anyhow::Result, and update all call sites to reference it via its fully-qualified path so it is never confused with std::result::Result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28refactor: add linternsfisis
2026-06-27refactor: fix compiler warnings and clippy warningsnsfisis
2026-06-26feat(http): reimplement CurlDownloader on reqwest; port 15 more testsnsfisis
Replace the libcurl-shim CurlDownloader with a reqwest+tokio implementation per the .ken sketch, resolving the construction panic that blocked command tests (mock path via __new_mock is untouched). Port remote_filesystem (7), hg/svn driver (4), zip_archiver/git_exclude_filter (4) tests. Fix hg/svn/git_exclude regex-delimiter and svn result-propagation porting bugs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26test: port 44 vcs/downloader/version tests using mock infransfisis
Port git, version_guesser, gitlab_driver, github_driver, and git_downloader tests using the ProcessExecutor/HttpDownloader mocks and IO/Config stubs. Fix production regex-porting bugs surfaced by the now-reachable paths: Url::sanitize and Response::find_header_value had non-delimited PCRE patterns; implement array_search_mixed non-strict branch and a datetime format mapping. Add HttpDownloader::__new_mock so mocked downloaders skip curl construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26feat(test): add HttpDownloaderMock, IOStub, and Config stub helpersnsfisis
IOStub and ConfigStubBuilder provide getMockBuilder-style configurable stubs. Wired into util/repository/downloader/command test targets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21refactor(math): use method-style max/min/clamp over std::cmpnsfisis
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20refactor(php-shim): drop Box wrapping from PhpMixed List/Arraynsfisis
The List and Array variants of PhpMixed boxed their elements unnecessarily. Store PhpMixed values directly and update all callers accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20refactor: auto-fix clippy warningsnsfisis
2026-06-14refactor(pcre): drop Result from Preg method return typesnsfisis
The Preg methods panic on PCRE failure (per the file header rationale), so their anyhow::Result wrappers never carried an Err. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14refactor(pcre): drop strict-groups Preg variantsnsfisis
Rust's type system already distinguishes participating from non-participating capture groups via Option, so the *StrictGroups methods add no safety here. Remove them and switch callers to the plain variants.
2026-06-10feat(phase-c): resolve cross-module phase-b TODOsnsfisis
2026-06-06refactor(http-response): take url directly instead of request mapnsfisis
Response only ever reads the url out of the request array, so accept it as a String directly. With url always present the 'url key missing' LogicException can no longer fire, so Response::new and CurlResponse::new return Self instead of a double Result. Also drops the unused from_php_mixed/to_php_mixed stubs and the request_to_map helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04feat: resolve trivial todo!() sites with existing constructorsnsfisis
Replace todo!("VersionParser::new()") with VersionParser::new() at 6 call sites (array_loader, base_command, create_project_command, vcs_repository, http_downloader x2) and EventDispatcher::io_clone with self.io.clone(). All targets already exist on their types. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03feat(downloader,repository): wire exception instanceof downcasts via anyhownsfisis
Resolve the PHP try/catch instanceof checks that select on exception type: TransportException-only catches in RepositorySet, SvnDriver and JsonFile; the RuntimeException/IrrecoverableDownloadException handleError closure and the RuntimeException update-retry catch in DownloadManager. Each uses anyhow::Error::downcast_ref and reads the exception's message field directly (Display is todo\!() on the shim exceptions). PHPUnit\Framework\Exception checks in VcsDownloader are documented as intentionally always-false since the test framework is out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29refactor(io): unify IOInterface params to Rc<RefCell<dyn _>>nsfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26refactor(io): share IOInterface via Rc<RefCell<dyn _>> handlensfisis
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23refactor(semver): change ConstraintInterface to a closed enumnsfisis
Replace the dyn ConstraintInterface trait objects with an AnyConstraint enum closing over its four implementors (Simple, Multi, MatchAll, MatchNone), mirroring the earlier Rule enum conversion. Rename constraint.rs to simple_constraint.rs to match the renamed Constraint type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23refactor(promise): drop \React\Promisensfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23refactor(promise): change functions returning PromiseInterface to async fnnsfisis
2026-05-22refactor: share Pool via Rc<RefCell>nsfisis
Convert Pool to Rc<RefCell<Pool>> so Solver, Decisions, and RuleSetGenerator share it, resolving the todo!() placeholders that blocked the dependency resolver (Phase C shared ownership). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22refactor(composer): unify Composer/PartialComposer via Rc handlesnsfisis
Model PHP's `Composer extends PartialComposer` as a PartialOrFullComposer enum and merge partial_composer.rs into composer.rs. Introduce ComposerHandle / PartialComposerHandle (plus their Weak variants) so the graph can be shared, and build it at once with Rc::new_cyclic in the factory to resolve the back-reference cycles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20refactor: re-export module items to shorten import pathsnsfisis
2026-05-20fix(compile): fix all remaining compile errorsnsfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19fix(compile): fix more random compile errorsnsfisis
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19fix(compile): fix various compile errorsnsfisis
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-17fix(shirabe): correct invalid import paths and add missing dependencynsfisis