Tiragolumab and Atezolizumab for the Treatment of Relapsed or Refractory SMARCB1 or SMARCA4 Deficient Tumors

Inclusion Criteria: * Patients must be \>= 12 months of age at the time of study enrollment. For part A, patients must be \< 18 years old at enrollment. For part B, there is no upper age limit * The Part B (phase 2) cohorts will initially open concurrently with the part A but will only enroll patients at least 18 years of age. Patients \< 18 years of age will be included in the part B cohorts only after the tiragolumab monotherapy dose has been assessed to be safe in the part A portion * Patients must have SMARCB1 (INI1) or SMARCA4 deficient tumors verified through institutional immunohistochemistry (IHC) or molecular confirmation of a pathologic SMARCB1 (INI1) or SMARCA4 loss or mutation in the tumor from a Clinical Laboratory Improvement Act (CLIA) certified lab with the following disease histologies: * Renal medullary carcinoma * Malignant rhabdoid tumor (extra-CNS) * Atypical teratoid rhabdoid tumor (CNS) * Poorly differentiated chordoma * Epithelioid sarcoma * Other SMARCB1 or SMARCA4 deficient tumors * Note: Molecular studies will only be used if IHC is equivocal or cannot be performed. Documentation of the institutional IHC or molecular testing must be uploaded via the RAVE system * Part A: Patients must have either measurable or evaluable disease Part B: Patients must have measurable disease per RECIST v1.1 for non-CNS tumors or CNS response criteria for CNS tumors * Note: See protocol for specific exclusion for patients with CNS primary or metastatic disease * Patients must have relapsed, refractory disease or newly diagnosed disease for which there is no known curative therapy or therapy proven to prolong survival with an acceptable quality of life * Patients must have a performance status corresponding to Eastern Cooperative Oncology Group (ECOG) scores of 0, 1 or 2 (Karnofsky/Lansky score of \>= 50). Use Karnofsky for patients \> 16 years of age and Lansky for patients =\< 16 years of age. Note: Neurologic deficits in patients with CNS tumors must have been stable for at least 7 days prior to study enrollment. Patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score * Patients must have fully recovered from the acute toxic effects of all prior anti-cancer therapy and must meet the following minimum duration from prior anti-cancer directed therapy prior to enrollment. If after the required timeframe, the numerical eligibility criteria are met, e.g., blood count criteria, the patient is considered to have recovered adequately * Cytotoxic chemotherapy or other anti-cancer agents known to be myelosuppressive: See Developmental Therapeutics (DVL) homepage on the Children's Oncology Group (COG) Members site for commercial and investigational agent classifications. For agents not listed, the duration of this interval must be discussed with the study chair and the study-assigned Research Coordinator prior to enrollment * \>= 21 days after the last dose of myelosuppressive chemotherapy (42 days if prior nitrosourea). Please refer to the table of myelosuppressive/Anticancer Agents on the COG website: https://www.cogmembers.org/uploadedFiles/Site/Disc/DVL/Documents/TableOfMyelosuppressiveAnti-CancerAgents.pdf * Anti-cancer agents not known to be myelosuppressive (e.g., not associated with reduced platelet or absolute neutrophil count \[ANC\] counts): \>= 7 days after the last dose of agent. See the DVL homepage on the COG Members site for commercial and investigational agent classifications. For agents not listed, the duration of this interval must be discussed with the study chair and the study-assigned Research Coordinator prior to enrollment * Antibodies: \>= 21 days must have elapsed from infusion of last dose of antibody, and toxicity related to prior antibody therapy must be recovered to grade =\< 1 * Hematopoietic growth factors: \>= 14 days after the last dose of a long-acting growth factor (e.g., pegfilgrastim) or 7 days for short acting growth factor. For agents that have known adverse events occurring beyond 7 days after administration, this period must be extended beyond the time during which adverse events are known to occur * Interleukins, interferons and cytokines (other than hematopoietic growth factors): \>= 21 days after the completion of interleukins, interferon or cytokines (other than hematopoietic growth factors) * Stem cell infusions (with or without total-body irradiation \[TBI\]): * Autologous stem cell infusion including boost infusion: \>= 30 days * Cellular therapy: \>= 30 days after the completion of any type of cellular therapy (e.g., modified T cells, natural killer \[NK\] cells, dendritic cells, etc.) * External radiation therapy (XRT)/external beam irradiation including protons: \>= 14 days after local XRT; \>= 90 days after TBI, craniospinal XRT or if radiation to \>= 50% of the pelvis; \>= 42 days if other substantial bone marrow (BM) radiation * Radiopharmaceutical therapy (e.g., radiolabeled antibody, iodine I 131 metaiodobenzylguanidine \[131I MIBG\]): \>= 42 days after systemically administered radiopharmaceutical therapy * Patients must not have had prior TIGIT targeting therapy * Patients must not have received prior therapy with an anti- PD-1, anti-PD-L1, anti-PD-L2, or anti-CTLA4 agent or with an agent directed to another stimulatory or co-inhibitory T cell receptor (i.e. OX-40, CD137) * Patients must not have received live/attenuated vaccine within 30 days of first dose of treatment * Patients must not be receiving concomitant systemic steroid medications and \>= 14 days must have elapsed since last dose of systemic corticosteroid with the following exceptions: * The use of physiologic doses of corticosteroids (5 mg/m\^2/day up to 10 mg/day of prednisone equivalent) is acceptable * The use of topical, inhaled, or ophthalmic corticosteroids are acceptable * The use of acute, low-dose systemic immunosuppressant medication or a one-time pulse dose of systemic immunosuppressant medication (e.g., 48 hours of corticosteroids for a contrast allergy) are acceptable * Treatment with systemic immunosuppressive medication (including, but not limited to, cyclophosphamide, azathioprine, methotrexate, thalidomide, and anti-tumor necrosis factor-alpha \[TNF-alpha\] agents) must have concluded \>= 14 days prior to study enrollment * For patients with solid tumors without known bone marrow involvement * Peripheral absolute neutrophil count (ANC) \>= 1000/uL (must be performed within 7 days prior to enrollment) * For patients with solid tumors without known bone marrow involvement * Platelet count \>= 100,000/uL (transfusion independent, defined as not receiving platelet transfusions for at least 7 days prior to enrollment) (must be performed within 7 days prior to enrollment) * Patients with known bone marrow metastatic disease will be eligible for study provided they meet the blood counts above (may receive transfusions provided they are not known to be refractory to red cell or platelet transfusions). These patients will not be evaluable for hematologic toxicity * A creatinine based on age/sex as follows (must be performed within 7 days prior to enrollment): * Age; Maximum Serum Creatinine (mg/dL) * 1 to \< 2 years; Male: 0.6; Female: 0.6 * 2 to \< 6 years; Male: 0.8; Female: 0.8 * 6 to \< 10 years; Male: 1; Female: 1 * 10 to \< 13 years; Male: 1.2; Female: 1.2 * 13 to \< 16 years; Male: 1.5; Female: 1.4 * \>= 16 years; Male: 1.7; Female: 1.4 OR- a 24 hour urine creatinine clearance \>= 70 mL/min/1.73 m\^2 (must be performed within 7 days prior to enrollment) OR- a glomerular filtration rate (GFR) \>= 70 mL/min/1.73 m\^2. GFR must be performed using direct measurement with a nuclear blood sampling method OR direct small molecule clearance method (iothalamate or other molecule per institutional standard) (must be performed within 7 days prior to enrollment) * Note: Estimated GFR (eGFR) from serum creatinine, cystatin C or other estimates are not acceptable for determining eligibility * Bilirubin (sum of conjugated + unconjugated or total) =\< 1.5 x upper limit of normal (ULN) for age (must be performed within 7 days prior to enrollment) * Patients with known Gilbert disease: Total bilirubin =\< 3 x ULN * Serum glutamic pyruvic transaminase (SGPT) (alanine aminotransferase \[ALT\]) =\< 135 U/L (must be performed within 7 days prior to enrollment). For the purpose of this study, the ULN for SGPT is 45 U/L * Albumin \>= 2 g/dL (must be performed within 7 days prior to enrollment) * Patients with seizure disorder may be enrolled if on anticonvulsants and well controlled as evidenced by no increase in seizure frequency in the prior 7 days * Nervous system disorders (Common Terminology Criteria for Adverse Events \[CTCAE\] v5) resulting from prior therapy must be =\< grade 2, with the exception of decreased tendon reflex (DTR). Any grade of DTR is eligible * International normalized ratio (INR) =\< 1.5 (must be performed within 7 days prior to enrollment) * Serum amylase =\< 1.5 x ULN (must be performed within 7 days prior to enrollment) * Serum lipase =\< 1.5 x ULN (must be performed within 7 days prior to enrollment) * Grade 1 or lower calcium level * Note: can have history of hypercalcemia as long as controlled and asymptomatic Exclusion Criteria: * Pregnant or breast-feeding women will not be entered on this study due to risks of fetal and teratogenic adverse events as seen in animal/human studies, OR because there is yet no available information regarding human fetal or teratogenic toxicities. Pregnancy tests must be obtained in female patients of childbearing potential. Female patients of childbearing potential are defined as those who are past the onset of menarche and are not surgically sterile (i.e., bilateral salpingectomy, bilateral oophorectomy, complete hysterectomy) or post-menopausal. Males or females of reproductive potential may not participate unless they have agreed to use two effective methods of birth control, including a medically accepted barrier or contraceptive method (e.g., male or female condom) for the duration of therapy and at least 90 days after final dose of tiragolumab and 150 days after final dose of atezolizumab, whichever is later. Abstinence is an acceptable method of birth control. * It is not known if atezolizumab or tiragolumab are present in breast milk; however, IgG immunoglobulins are found in milk. Due to the potential for serious adverse reactions in the breastfed infant, breastfeeding is not recommended during therapy and for at least 150 days after the last dose of atezolizumab and 90 days after the last dose of tiragolumab, whichever is later * Concomitant medications: * Corticosteroids: * Patients must not be receiving concomitant systemic steroid medications and \>= 14 days must have elapsed since last dose of systemic corticosteroid with the following exceptions: * The use of physiologic doses of corticosteroids (5 mg/m\^2/day up to 10 mg/day of prednisone equivalent) is acceptable * The use of topical, inhaled, or ophthalmic corticosteroids are acceptable * The use of acute, low-dose systemic immunosuppressant medication or a one-time pulse dose of systemic immunosuppressant medication (e.g. 48 hours of corticosteroids for a contrast allergy) are acceptable * Investigational drugs: Patients who are currently receiving another investigational drug are not eligible * Anti-cancer Agents: Patients who are currently receiving other anti-cancer agents are not eligible * Systemic immunosuppressive medications (including, but not limited to, cyclophosphamide, azathioprine, methotrexate, and thalidomide) during study treatment because these agents could potentially alter the efficacy and safety of study treatments would not be eligible * Patients must not have a known hypersensitivity to any component of tiragolumab or atezolizumab injection * History of severe allergic anaphylactic reactions to chimeric or humanized antibodies or fusion proteins * Known hypersensitivity to Chinese hamster ovary cell products or to any component of the atezolizumab or tiragolumab formulation * Patients who have undergone allogeneic bone marrow or allogeneic cell transplant are not eligible * Patients with CNS metastases from non-CNS primary tumors are not eligible unless CNS metastases have been previously treated and sequential imaging shows no evidence for active disease in the CNS. * Patients with primary CNS tumors (including ATRT) with involvement of the brainstem are not eligible. Note: Patients with ATRT with M0-M4 disease without involvement of the brain stem are allowed to participate * Patients must not have active autoimmune disease that has required systemic treatment in the past 12 months, or a documented history of clinically severe autoimmune disease, or a syndrome that requires systemic steroids or immunosuppressive agents. Subjects with vitiligo or resolved childhood asthma/atopy are not excluded. Replacement therapy (e.g. thyroxine, insulin, physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment and these patients are eligible * Patients who have active immune deficiency are not eligible * Patients who have known active tuberculosis are not eligible * Hepatitis B or C infection: * Patients \< 18 years old at enrollment, who have known hepatitis B or C * Patients \>= 18 years old at enrollment with: * Positive hepatitis B surface antigen (HBsAg), OR * Positive total hepatitis B core antibody (HBcAb) who have a quantitative hepatitis B virus (HBV) deoxyribonucleic acid (DNA) \>= 500 IU/mL, OR * Positive hepatitis C virus (HCV) antibody with a positive HCV ribonucleic acid (RNA) test * Note: For adults (\>= 18 years old at enrollment), hepatitis B serology testing is required to determine eligibility. The HBV DNA test is required only for patients who have a negative HBsAg test, a negative HBsAb test, and a positive total HBcAb test. For adults (\>= 18 years old at enrollment), hepatitis C serology testing is required to determine eligibility. The HCV RNA test is required only for patients who have a positive HCV antibody test * Patients who have a known, recent Epstein-Barr virus (EBV) infection or known history of chronic, active infection are not eligible * Patients who have history of or active human immunodeficiency virus (HIV) are not eligible except patients who are stable on anti-retroviral therapy, have a CD4 count \>= 200/uL, and have an undetectable viral load * Patients who have significant cardiovascular disease (such as New York Heart Association class III or IV congestive heart failure, myocardial infarction, or cerebrovascular accident) within 3 months prior to study enrollment, unstable arrhythmia, or unstable angina are not eligible * Patients who have a major surgical procedure, other than for diagnosis, within 4 weeks prior to study enrollment, or the anticipation of the need for a major surgical procedure during the study are not eligible * Patients who have a history of idiopathic pulmonary fibrosis, organizing pneumonia, drug-induced pneumonitis, idiopathic pneumonitis, or known active pneumonitis are not eligible. History of radiation pneumonitis in the radiation field is permitted * Patients who have uncontrolled pleural effusion, pericardial effusion, or ascites requiring recurrent drainage procedures (once monthly or more frequently) are not eligible. Patients with indwelling catheters (e.g., PleurX) are allowed * Patients who have an uncontrolled infection are not eligible * Patients who have received a prior solid organ transplantation are not eligible * Patients who in the opinion of the investigator may not be able to comply with the safety monitoring requirements of the study are not eligible