Testing the Addition of Immunotherapy With Hu5F9-G4 (Magrolimab) to the Usual PARP Inhibitor, Olaparib for Treatment of Metastatic or Recurrent Breast or Castrate-Resistant Prostate Cancer With BRCA Mutations

Inclusion Criteria: * Presence of metastatic and/or recurrent solid tumors (therapy naïve or those with prior therapy) with pathogenic BRCA 1/2 mutated cancers where olaparib is indicated as standard of care therapeutic option (not maintenance) as below: * Metastatic breast cancer: * Germline - Yes (Y); Somatic - No (N): Must have had prior chemotherapy (chemo) (adjuvant \[adj\], treatment); or if hormone receptor positive (HR+), must have had prior endocrine therapy or deemed inappropriate for endocrine treatment * Metastatic castrate-resistant prostate cancer (CPRC) * Germline - Y; Somatic - Y: Treatment after progressive disease on anti-androgens * BRCA mutation status must be confirmed in a Clinical Laboratory Improvement Act (CLIA)-certified laboratory (lab) * Patients \>= 18 years of age. Because no dosing or adverse event data are currently available on the use of Hu5F9-G4 (magrolimab) in combination with olaparib in patients \< 18 years of age, children are excluded from this study * Patients may not have had prior PARP inhibitor in the metastatic setting when given for therapeutic purposes. Patients with breast cancer who received adjuvant PARP inhibitor (i) are eligible. Patients who had prior maintenance therapy are eligible. * At least 4 weeks or 4 prior drug half-lives (whichever is shorter) must have elapsed since completion of the previous systemic therapy * Expansion Cohort: Willingness and feasibility to undergo pre and post treatment biopsy * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%) * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) * Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN * Glomerular filtration rate (GFR) \>= 50 mL/min/1.73 m\^2 unless data exists supporting safe use at lower kidney function values, no lower than 30 mL/min/1.73 m\^2 * Hemoglobin (Hgb) \>= 9 g/dL * Patients infected with human immunodeficiency virus (HIV) who are on effective anti-retroviral therapy with undetectable viral load within 6 months may be eligible for this trial * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load * Patients with treated brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression. Patients with prior brain metastasis that was treated with evidence of resolution or stable disease for 6 months are eligible. Patients are required to be stable and fully tapered off steroids for at least \>= 1 month * Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial * Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better * Women of child-bearing potential, and men who are sexually active and their partner who may become pregnant, must use contraception during treatment and at least for 6 months (women) and 4 months (men) after the last dose of magrolimab and olaparib. Breastfeeding is not allowed during treatment and for one month after receiving the last dose of therapy * Ability to understand and the willingness to sign a written informed consent document. Legally authorized representatives may sign and give informed consent on behalf of study participants. Exclusion Criteria: * Previous anti-CD47 therapy * Concomitant use of strong CYP3A4 inhibitors/inducers can cause clinically significant drug-interactions; thus, study patients who require the use of these CYP enzymes continuously should be excluded. Study patients need to come off 3 eliminated half-lives of moderate CYP3A4 inhibitors and 5 eliminated half-lives of strong CYP3A4 inhibitors * Patients who require immunosuppressive treatments for comorbidities are not eligible * Gastrointestinal pathology or history that adversely impacts the ability to take or absorb oral medication * Inability to comply with the protocol and/or not willing or who will not be available for follow-up assessments * Patients who are receiving any other investigational agents * History of allergic reactions attributed to compounds of similar chemical or biologic composition to magrolimab or other agents used in study * Patients with uncontrolled intercurrent illness * Pregnant women are excluded from this study because Hu5F9-G4 (magrolimab) is an anti-CD47 monoclonal antibody agent with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with Hu5F9-G4 (magrolimab), breastfeeding should be discontinued if the mother is treated with Hu5F9-G4 (magrolimab). These potential risks may also apply to other agents used in this study