Avelumab and M6620 for the Treatment of DDR Deficient Metastatic or Unresectable Solid Tumors

Inclusion Criteria: * Subjects must have histologically confirmed malignancy that is metastatic or unresectable and for which standard curative measures do not exist or are no longer effective * Subjects will be eligible for this study based on the presence of actionable aberrations in one or more of the following DNA damage response (DDR) genes: ARID1A, ATM, ATR, ATRX, BAP1, BARD1, BRCA1/2, BRIP1, CDK12, CHEK2, FANCA, FANCC, FANCD2, FANCE, FANCF, FANCM, MRE11A, MSH2, NBN (NBS1), PALB2, RAD51, RAD51C, RAD51D, SMARCB1, and VHL, or other related genes at the discretion of the principal investigator in consultation with the MD Anderson Cancer Center Institute for Personalized Cancer Therapy Precision Oncology Decision Support (PODS) group. Variant interpretation for actionability will be performed by PODS * Subjects with germline defects in DDR genes are eligible for this trial * The collection of archival tumor tissue (within 1 year prior to study enrollment) will be mandatory. Tumor biopsies on cycle 1 day 15 will be mandatory. Subjects will not be put at undue risk to obtain the biopsy at cycle 1 day 15 (C1D15). A biopsy at C1D15 will not be required if it poses a serious/severe complication risk greater than 2%. All other biopsy time points are not mandatory but will be strongly encouraged where feasible. These include at baseline and at disease progression. Archival and fresh tissue requests can be waived in exceptional circumstances with principal investigator (PI) approval and only where rationale is documented * Subjects must have received at least 1 line of systemic therapy in the advanced/metastatic setting. Subjects with diseases without known effective options, and subjects who had declined standard of care therapy prior to study introduction are also eligible * Subjects enrolling in the dose escalation should have progressed on or be intolerant to all therapies known to confer a clinical benefit. Subjects must not have refused all available therapies * Subjects who have received prior therapy with immune checkpoint inhibitors (ICIs) are eligible for this trial. Subjects with a standard-of-care option for an immune checkpoint inhibitor are eligible * Subjects must have measurable disease per Response Evaluation Criteria in Solid Tumors (RECIST) version (v) 1.1, or patients may have bone metastatic disease evaluable by Prostate Cancer Working Group 3 (PCWG3) for subjects with metastatic castration-resistant prostate cancer (CRPC), or according to tumor evaluation criteria best suited and accepted for the tumor type to be evaluated * Subjects must have an Eastern Cooperative Oncology Group (ECOG) performance status 0-1 * Subjects must have a life expectancy \>= 12 weeks * Absolute neutrophil count \>= 1.5 x 10\^9/L * Platelets \>= 100 x 10\^9/L * Hemoglobin \>= 9 g/dL or \>= 5.6 mmol/L * Total bilirubin =\< 1.5 X the institutional upper limit of normal (ULN) * Aspartate aminotransferase (AST) serum glutamic-oxaloacetic transaminase (SGOT)/ alanine aminotransferase (ALT) serum glutamic-pyruvic transaminase (SGPT) =\< 2.5 X institutional ULN or =\< 5 X institutional ULN in the presence of liver metastases * Serum creatinine =\< 2 X ULN or estimated creatinine clearance \>= 30 mL/min according to the Cockcroft-Gault formula * Female patients of childbearing potential must have a negative serum or urine pregnancy test at screening (and again at baseline just prior to first administration of study drugs) * Female patients of non-childbearing potential must meet at least 1 of the following criteria: * Achieved postmenopausal status, defined as follows: cessation of regular menses for at least 12 consecutive months, with no alternative pathological or physiological cause; * Have undergone a documented hysterectomy and/or bilateral oophorectomy * Have medically confirmed ovarian failure All other female patients (including female patients with tubal ligation) are considered to be of childbearing potential * Women of childbearing potential and fertile men must agree to use adequate contraception when sexually active from signing of the informed consent form for the full study until at least 6 months after the last study drug administration. Patients must agree to utilize 2 reliable and acceptable methods of contraception simultaneously. A man is considered fertile after puberty unless permanently sterile by bilateral orchiectomy. Men taking part in this study are advised not to father a child during and up to 6 months after treatment; prior to treatment, advice should be sought for conserving sperm due to the chance of irreversible infertility as a consequence of treatment. Female partners of childbearing potential from male study patients have to use adequate contraception / birth control between signing of the informed consent and 6 months after the last administration of the study drug if the male study patient is not sterilized. The investigator or a designated associate is requested to advise the patient how to achieve highly effective birth control. Highly effective (failure rate of less than 1% per year) contraception methods, when used consistently and correctly, include: * Combined (estrogen and progestin containing: oral, intravaginal, transdermal) and progestin-only (oral, injectable, implantable) hormonal contraception associated with inhibition of ovulation * Intra-uterine device or intrauterine hormone-releasing system * Bilateral tubal occlusion or vasectomized partner (provided that partner is the sole sexual partner and has received medical assessment of the surgical success) * Sexual abstinence (reliability to be evaluated in relation to the duration of the clinical trial and the preferred and usual lifestyle of the patient) Male patients with a female partner of reproductive potential must use a condom and ensure that an additional form of contraception is also used during treatment and until 6 months after last study drug administration. Patients must agree to utilize 2 reliable and acceptable methods of contraception simultaneously * Evidence of a personally signed and dated informed consent document indicating that the patient has been informed of all pertinent aspects of the study * Willing and able to comply with scheduled visits, treatment plan, laboratory tests, and other procedures * Human immunodeficiency virus (HIV)-infected (HIV1/2 antibody-positive) patients may participate IF they meet all the following eligibility requirements: * They must be on an anti-retroviral regimen with evidence of at least two undetectable viral loads within the past 6 months on this same regimen; the most recent undetectable viral load must be within the past 12 weeks * They must have a CD4 count \>= 250 cells/mcL over the past 6 months on this same anti-retroviral regimen and must not have had a CD4 count \< 200 cells/ mcL over the past 2 years, unless it was deemed related to the cancer and/or chemotherapy-induced bone marrow suppression * For patients who have received chemotherapy in the past 6 months, a CD4 count \< 250 cells/mcL during chemotherapy is permitted as long as viral loads were undetectable during this same chemotherapy * They must have an undetectable viral load and a CD4 count \>= 250 cells/mcL within 7 days of enrolment * They must not be currently receiving prophylactic therapy for an opportunistic infection and must not have had an opportunistic infection within the past 6 months Exclusion Criteria: * Anticancer systemic therapy or radiotherapy within 4 weeks or 5 half-lives, whichever is shorter prior to starting the study agents. Prior palliative radiotherapy to metastatic lesion(s) is permitted, provided it has been completed 2 weeks prior to study enrollment, and no clinically significant toxicities are expected (e.g., mucositis, esophagitis) * Known symptomatic brain metastases requiring steroids. Patients with previously treated diagnosed brain metastases are eligible if they have completed their treatment and have recovered from the acute effects of radiation therapy or surgery prior to study enrollment, have discontinued corticosteroid treatment for these metastases for at least 4 weeks and are neurologically stable * Current use of immunosuppressive medication at the time of study enrollment, EXCEPT for the following permitted steroids: * Intranasal, inhaled, topical steroids, eye drops, or local steroid injection (e.g., intra-articular injection) * Systemic corticosteroids at physiologic doses =\< 10 mg/day of prednisone or equivalent * Steroids as premedication for hypersensitivity reactions (e.g. computed tomography \[CT\] scan premedication) * Subjects who had major surgery within 4 weeks prior to study enrollment * Known prior severe hypersensitivity to investigational products or any component in their formulations, including known severe hypersensitivity reactions to monoclonal antibodies (National Cancer Institute \[NCI\] Common Terminology Criteria for Adverse Events \[CTCAE\] version \[v\] 5 grade \>= 3) * Active infection requiring systemic therapy * Known history of immune-mediated colitis, inflammatory bowel disease, pneumonitis, and pulmonary fibrosis * Active or prior autoimmune disease that may deteriorate when receiving an immunostimulatory agent. Patients with type I diabetes, vitiligo, psoriasis, or hypo- or hyperthyroid disease not requiring immunosuppressive treatment are eligible * Prior organ transplantation including allogenic stem cell transplantation * Diagnosis of myelodysplastic syndrome (MDS) * Vaccination within 4 weeks of study enrollment and while on trial is prohibited except for the administration of inactivated vaccines * Clinically significant (i.e., active) cardiovascular disease: cerebral vascular accident/stroke (\< 6 month prior to enrollment), myocardial infarction (\< 6 months prior to enrollment), unstable angina, congestive heart failure (\>= New York Heart Association Classification class II) or a serious cardiac arrhythmia requiring medication * Other acute or chronic medical or psychiatric conditions including but not limited to recent (within the past year) or active suicidal ideation or behavior or laboratory abnormality that may increase the risk associated with study participation or investigational product administration or may interfere with the interpretation of study results, and in the judgement of the Investigator, would make the patient inappropriate for entry into this study * Pregnant female patients, breastfeeding female patients, fertile male patients, and female patients of childbearing potential who are unwilling or unable to use 2 highly effective methods of contraception as outlined in this protocol for the duration of the study, and for at least 6 months after the last dose of study drug administration * Hepatitis B virus (HBV) or hepatitis C virus (HCV) infection at screening (positive HBV surface antigen or HCV ribonucleic acid (RNA) if anti-HCV antibody screening test is positive) * Known additional malignancy that is active and/or progressive requiring treatment; exceptions include basal cell or squamous cell skin cancer, in situ bladder cancer, or other cancer for which the patient has been disease-free for \>= 2 years * Persisting toxicity related to prior therapy (NCI CTCAE v5 grade \> 1); however, alopecia and sensory neuropathy grade =\< 2, or other grade =\< 2 AEs not constituting a safety risk, based on the investigator's judgement, are acceptable * Subjects receiving treatment with strong inhibitors or inducers of CYP3A4 that cannot be discontinued before start of investigational treatment and for the duration of study * Subjects with ongoing toxicity (any grade) and/or resolved ICI toxicity (grade 3 or higher only)