Breast Cancer Vaccine in Combination With Pembrolizumab for Treatment of Persistent, Recurrent, or Metastatic Breast Cancer

Inclusion Criteria: * Have evidence of persistent, recurrent, or progressive metastatic breast cancer for which there is no known or established treatment available with curative intent, after failing at least two courses of community standard systemic treatment with chemotherapy (and endocrine therapy, if appropriate) * Human epidermal growth factor 2 (HER2) positive and estrogen receptor (ER) or progesterone receptor (PR) positive tumors: must be refractory to hormonal therapy (e.g., aromatase inhibitor, tamoxifen or fulvestrant) and previously treated with at least 1 regimen that includes at least two antiHER2 agents (e.g., trastuzumab and pertuzumab) * ER/PR positive, HER2 negative tumors: must be refractory to hormonal therapy (e.g. aromatase inhibitor, tamoxifen or fulvestrant) and previously treated with at least 2 chemotherapy containing regimens * HER2 positive and ER/PR negative tumors: must have failed at least 2 regimens including at least two anti-HER2 agents (e.g., trastuzumab and pertuzumab) * Triple negative tumors (defined as ER \< 1%, PR \< 1% as per American Society of Clinical Oncology College of American Pathologists \[ASCO CAP\] guidelines, HER2/neu 0-1 by immunohistochemistry \[IHC\] or negative by dual in situ hybridization \[ISH\]): Must have failed two other treatment lines including either a taxane and/or atezolizumab. Patients can have had atezolizumab (PD-L1 inhibitor) or PD-1 therapy previously * Patients will only be eligible for this study if they have at least one human leukocyte antigen (HLA) match: * HLA-A\*24:02 * HLA-B\*35:08 * HLA-B\*55:01 * HLA-C\*04:01 * HLA-C\*01:02 * HLA-DRB3\*01:01 * HLA-DRB3\*02:02 * Have expected survival of at least 4 months * Have adequate performance status (Eastern Cooperative Oncology Group \[ECOG\] 0-1) * A female participant is eligible to participate if she is not pregnant, not breastfeeding, and at least one of the following conditions applies: * Not a woman of childbearing potential (WOCBP) defined as spontaneous cessation of menstrual cycle for at least 12 months or surgical history of hysterectomy or bilateral salpingoopherectomy OR * A WOCBP who agrees to take appropriate precautions to avoid becoming pregnant during the treatment period and for at least 90 days plus 30 days (a menstruation cycle) for study treatments with risk of genotoxicity after the last dose of study treatment * The participant (or legally acceptable representative if applicable) provides written informed consent for the trial * Have measurable disease based on Response Evaluation Criteria in Solid Tumors (RECIST) 1.1. Lesions situated in a previously irradiated area are considered measurable if progression has been demonstrated in such lesions * If any brain metastases, must have had prior radiation therapy for brain metastases * Systemic glucocorticoids for any purpose other than to modulate symptoms from an event of clinical interest of suspected immunologic etiology are allowed. The use of physiologic doses of corticosteroids may be approved after consultation with the principal investigator * Absolute neutrophil count (ANC) \>= 1500/uL (collected within 14 days prior to the start of study treatment) * Platelets \>= 100,000/uL (collected within 14 days prior to the start of study treatment) * Hemoglobin \>= 9.0 g/dL or \>= 5.6 mmol/L (collected within 14 days prior to the start of study treatment) \* Criteria must be met without erythropoietin dependency and without packed red blood cell (pRBC) transfusion within last 2 weeks * Creatinine =\< 1.5 x upper limit of normal (ULN) OR measured or calculated creatinine clearance (glomerular filtration rate \[GFR\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) \>= 30 mL/min for participant with creatinine levels \> 1.5 x institutional ULN (collected within 14 days prior to the start of study treatment) \* Creatinine clearance (CrCl) should be calculated per institutional standard * Total bilirubin =\< 1.5 x ULN OR direct bilirubin =\< ULN for participants with total bilirubin levels \> 1.5 ULN * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase) \[\[SGOT\]) and alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase) \[\[SGPT\]) =\< 2.5 x ULN (=\< 5 x ULN for participants with liver metastases) (collected within 14 days prior to the start of study treatment) * International normalized ratio (INR) OR prothrombin time (PT) =\< 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or activated partial thromboplastin time (aPTT) is within therapeutic range of intended use of anticoagulants (collected within 14 days prior to the start of study treatment) * Activated partial thromboplastin time (aPTT) =\< 1.5 x ULN unless participant is receiving anticoagulant therapy as long as PT or aPTT is within therapeutic range of intended use of anticoagulants (collected within 14 days prior to the start of study treatment) Exclusion Criteria: * Concurrent or recent chemotherapy, radiotherapy, immunotherapy, or general anesthesia/major surgery within 4 weeks * Patients must have recovered from all known or expected toxicities from previous treatment and passed a treatment-free "washout" period of 3 weeks before starting this program * If participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting study treatment * Has received prior systemic anti-cancer therapy including investigational agents within 4 weeks prior to enrollment * Participants must have recovered from all adverse events (AEs) due to previous therapies to =\< grade 1 or baseline \* Participants with =\< grade 2 neuropathy and/or alopecia may be eligible * Has received a live vaccine within 30 days prior to the first dose of study drug. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, varicella/zoster (chicken pox), yellow fever, rabies, Bacillus Calmette-Guerin (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g, FluMist) are live attenuated vaccines and are not allowed * Has known active central nervous system (CNS) metastases and/or carcinomatous meningitis. Participants with previously treated brain metastases may participate provided they are radiologically stable, i.e. without evidence of progression for at least 4 weeks by repeat imaging (note that the repeat imaging should be performed during study screening), clinically stable and without requirement of steroid treatment for at least 14 days prior to first dose of study treatment * History of clinical hypersensitivity to granulocyte-macrophage colony-stimulating factor (GM-CSF), interferon-alpha-2b, yeast, beef, or to any components used in the preparation of SV-BR-1-GM * Has severe hypersensitivity (\>= grade 3) to pembrolizumab and/or any of its excipients * Proteinuria \>1+ on urinalysis or \>1 gm/24hr * Left ventricular ejection fraction (LVEF as determined by cardiac echo or multigated acquisition scan \[MUGA\] scan) below the normal limits of the institutions' specific testing range. This assessment may be repeated once at the discretion of the Investigator with the approval of the principal investigator * New York Heart Association stage 3 or 4 cardiac disease * A pleural or pericardial effusion of moderate severity or worse * WOCBP who have a positive urine pregnancy test within 7 days prior to enrollment \* If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Women who are pregnant or nursing * Patients with concurrent second malignancy. Persons with previous malignancies effectively treated and not requiring treatment for \> 24 months are eligible, provided there is unambiguous documentation that current local recurrence or metastatic site represents recurrence of the primary breast malignancy * Patients who are human immunodeficiency virus (HIV) positive (by self-report) or have clinical or laboratory features indicative of acquired immunodeficiency syndrome (AIDS) * Has a known history of hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (defined as HCV ribonucleic acid \[RNA\] is detected) infection * Has a history of (non-infectious) pneumonitis that required steroids or has current pneumonitis * Has an active infection requiring systemic therapy * Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g. thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment * Patients who require systemic steroids at doses \> 10 mg daily of prednisone or equivalent or any immunosuppressive drugs. Beta-blocker therapy, while not exclusionary, is discouraged and alternatives should be sought if possible. The beta-blocker might compromise use of epinephrine for the rare possibility of anaphylaxis * Patients with a history of colitis * Patients with severe psychiatric (e.g., schizophrenia, bipolar, or borderline personality disorder) or other clinically progressive major medical problems, unless approved by the principal investigator * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the study, interfere with the subject's participation for the full duration of the study, or is not in the best interest of the subject to participate, in the opinion of the treating investigator * Patients may not be on a concurrent treatment clinical trial, unless approved by principal investigator